+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190706/B0b --output-run DEEP/20190706/B0b/science#step1/20240310T152300Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 2.0749 seconds; current memory usage: 0.179 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0072 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0091 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/pipeline_orig.qgraph DEEP/20190706/B0b/science#step1/20240310T152300Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 7.3075 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 19.0661 seconds; current memory usage: 0.364 Gibyte, delta: 0.184 Gibyte, peak delta: 0.172 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.364 Gibyte (main), 0.351 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0274 seconds; current memory usage: 0.364 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.364 Gibyte (main), 0.351 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 558 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190706_B0b_science#step1_20240310T152300Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.1766 seconds; current memory usage: 0.364 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.364 Gibyte (main), 0.351 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 560 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2764 seconds; current memory usage: 0.365 Gibyte, delta: 0.001 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.365 Gibyte (main), 0.351 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 2ce9663a-dd70-4c1d-be53-bd40a9dfefe0 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-10 15:23:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-10 15:23:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 15:23:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-10 15:23:51 proc_lsst.multi:170 [INFO] [multi] job_id 90235 proc_lsst.multi INFO: [multi] job_id 90235 2024-03-10 15:23:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:51 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 90235 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 90235 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 90235 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash a180f3a1805e10b0314333669c3f8942 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8be340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/8a27ee53-5259-472c-b5dc-0e40c610167b_isr_871042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/8a27ee53-5259-472c-b5dc-0e40c610167b_isr_871042_5.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash b71d270bd82b51860034c525141f4233 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c25dae18-a6eb-4420-8dfa-f2b3b60ed6cf_isr_871041_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c25dae18-a6eb-4420-8dfa-f2b3b60ed6cf_isr_871041_53.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 9191463489bcf1af8624fdd0622630b7 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d48ee22-2fb6-4c8c-a384-306fb442538d_isr_871042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d48ee22-2fb6-4c8c-a384-306fb442538d_isr_871042_56.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash fd29caf3d743e400a80e56e1936fd225 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/96a8a182-5b74-4732-82ad-76c8aeb72e54_isr_871043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/96a8a182-5b74-4732-82ad-76c8aeb72e54_isr_871043_2.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 4b614491c565d508723c63433e8aa779 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/db32f59b-076d-49c6-961e-5392169299a3_isr_871041_33.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/db32f59b-076d-49c6-961e-5392169299a3_isr_871041_33.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash fa81fb2f692844ae2fc8349a4d375469 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9d265c59-223a-4931-9894-e4e0a55f7660_isr_871043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9d265c59-223a-4931-9894-e4e0a55f7660_isr_871043_58.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 9e51250acdf07e281c2636a2b4dd1888 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/93cbfb0b-ac12-4235-85a0-f94b79493654_isr_871043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/93cbfb0b-ac12-4235-85a0-f94b79493654_isr_871043_13.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 8220b3f93df738efac10ad8f5829b288 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/73d3f025-5968-44cd-a321-81eb7972499d_isr_871041_54.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/73d3f025-5968-44cd-a321-81eb7972499d_isr_871041_54.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash b16ad4033c7ac9b74e403b69435c2cbb parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e4b0da5d-a8e0-4c26-a978-bcdf0c644ebf_isr_871041_13.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e4b0da5d-a8e0-4c26-a978-bcdf0c644ebf_isr_871041_13.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 8c48e4a460f2917aaa55fd4fc3132ff3 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/fb96685d-5500-4826-836b-1856f61eeec2_isr_871043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/fb96685d-5500-4826-836b-1856f61eeec2_isr_871043_21.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash bea3ef6ae242ece06040fa3385ba0be4 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2baf73b3-5b33-494c-a22b-9eb6d594a5d2_isr_871043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2baf73b3-5b33-494c-a22b-9eb6d594a5d2_isr_871043_30.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 5ebec140d954ee8a4d8b5a3c0e86efde parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/6a1bb29e-a896-49c2-b8e7-e9cfa01ffa23_isr_871042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/6a1bb29e-a896-49c2-b8e7-e9cfa01ffa23_isr_871042_31.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 5fc7405453807d4a0bd0ba1f21f3a752 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dcfdaa38-5861-42f1-8d82-ce5fb7c0689b_isr_871042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dcfdaa38-5861-42f1-8d82-ce5fb7c0689b_isr_871042_42.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 169da8398c481bb17c255b82e97dc863 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/cec61354-9d52-4f11-8286-8ca7bddfdfa2_isr_871043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/cec61354-9d52-4f11-8286-8ca7bddfdfa2_isr_871043_4.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash baeb27904080355cbd4795640c6f1817 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c6393991-0ddb-4ece-b520-c692dfa501cd_isr_871041_56.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c6393991-0ddb-4ece-b520-c692dfa501cd_isr_871041_56.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 486c7a956206e4c89493184bbbd0c711 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/936b6d74-ba49-492b-9557-95ee4cd0ef07_isr_871041_57.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/936b6d74-ba49-492b-9557-95ee4cd0ef07_isr_871041_57.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 4e5bf69c322cf3477817a12254cee36e parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf8a4251-3f7a-4325-9eb9-b2dee2a4472d_isr_871041_26.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf8a4251-3f7a-4325-9eb9-b2dee2a4472d_isr_871041_26.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 7745f9ff40887c645b654f530238a9cf parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9a003aa3-8ca4-46ae-83db-977823e9caef_isr_871042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9a003aa3-8ca4-46ae-83db-977823e9caef_isr_871042_8.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 7bddae7a7563ee362f6d85b2ecb3dcee parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/777caa82-45bb-4d1a-8cc3-97c7d5e197a2_isr_871042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/777caa82-45bb-4d1a-8cc3-97c7d5e197a2_isr_871042_57.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 8de7e41910e98e8f689a012183617763 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/768e0228-380c-4c92-adee-6a6a541933e5_isr_871041_18.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/768e0228-380c-4c92-adee-6a6a541933e5_isr_871041_18.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash e4e2d242a115d78bbe3d897f8cbb1288 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b6450d27-31b2-4450-9520-80117de54def_isr_871041_30.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b6450d27-31b2-4450-9520-80117de54def_isr_871041_30.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash bab9443b0f1259a00e7190ecfb4b9497 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4d8057da-dc69-42a8-9340-ff0b27659b58_isr_871042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4d8057da-dc69-42a8-9340-ff0b27659b58_isr_871042_11.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash c66e0a5ee9f55e25170a8330f210a0c7 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1bcc3775-1038-4232-ada5-24d6b1b2d71f_isr_871042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1bcc3775-1038-4232-ada5-24d6b1b2d71f_isr_871042_27.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash c66bc8f808cebdda7d576f67f424722e parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d2e389a-9ef2-4675-994d-f4c45a32e4f7_isr_871042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d2e389a-9ef2-4675-994d-f4c45a32e4f7_isr_871042_12.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 6377abcd603a86a125ce67c616b5d435 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/4e727cfe-2f09-4ed3-9d6a-01e099ae6212_isr_871043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/4e727cfe-2f09-4ed3-9d6a-01e099ae6212_isr_871043_49.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 4ce48325100388de12767d3ac5d96668 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a8bf1a77-e592-411c-b752-01ac151cafb6_isr_871042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a8bf1a77-e592-411c-b752-01ac151cafb6_isr_871042_33.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash b9f6923037126d354ca68b250da12ca3 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/1d7626b1-8278-499f-a7cf-e698f2b0bbc6_isr_871041_3.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/1d7626b1-8278-499f-a7cf-e698f2b0bbc6_isr_871041_3.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 38bb98bc0aa3c9f8a4ee1e366a4e14c6 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/bd18eca1-4347-4eaf-a3c6-655316112332_isr_871042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/bd18eca1-4347-4eaf-a3c6-655316112332_isr_871042_53.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash d2267179c3aac868a1302d7671f99e67 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9002c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/220ac616-7ce6-482b-8566-fbf110e53f91_isr_871043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/220ac616-7ce6-482b-8566-fbf110e53f91_isr_871043_22.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash b1848d19afeaf4d81f6c686aba1ad259 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9005e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ffa6a6ab-e8aa-4ee2-9170-76a7e10bb9ab_isr_871041_55.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ffa6a6ab-e8aa-4ee2-9170-76a7e10bb9ab_isr_871041_55.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 3deec5f03131ca0ecc3d69a6851e334a parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0f54af3c-ad64-447b-b04d-681a02a13d37_isr_871041_19.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0f54af3c-ad64-447b-b04d-681a02a13d37_isr_871041_19.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash d9ce81dbe224ef0c96048b8b12c315f9 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c9c2cd8d-2f35-4dc7-bc95-1b5c4334a2c5_isr_871042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c9c2cd8d-2f35-4dc7-bc95-1b5c4334a2c5_isr_871042_59.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 2fcb896409d215737ad1a9c0c94c8786 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a90f9e48-ffbf-4c92-8735-5deb9d84cc8b_isr_871042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a90f9e48-ffbf-4c92-8735-5deb9d84cc8b_isr_871042_62.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 85e67adb88071c9746c9c78a233e8490 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/d021aa0d-cbba-4373-a0c6-f421f70a4579_isr_871042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/d021aa0d-cbba-4373-a0c6-f421f70a4579_isr_871042_16.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash e77336d4d5ff4c69ba2b3820024b38b0 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dcf6c372-39e7-48c9-9707-c15e538602cc_isr_871043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dcf6c372-39e7-48c9-9707-c15e538602cc_isr_871043_15.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 54a78ba5a5028f5cd424d103e3b63e5c parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2220fb4-61e9-4f3d-b190-70e4dbd89726_isr_871042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2220fb4-61e9-4f3d-b190-70e4dbd89726_isr_871042_13.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash bba1ab064568dea175238effdfc0cb14 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a08f3702-9d9c-4fe6-ba93-d117c16831f5_isr_871043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a08f3702-9d9c-4fe6-ba93-d117c16831f5_isr_871043_50.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 49d12a0043d6c4e1a0c62e996d4ba8cc parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/23017e1b-098c-4824-99fb-9d0f46a0e82d_isr_871043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/23017e1b-098c-4824-99fb-9d0f46a0e82d_isr_871043_37.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash c0936358bb241787955b8f3c48e0502e parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2396315e-2062-4ffd-8606-00bb3601718b_isr_871043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2396315e-2062-4ffd-8606-00bb3601718b_isr_871043_48.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash c58cd0675edf820db9fd59fb14c17c2b parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e33a43c-21e8-4446-9759-2f33c634773e_isr_871043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e33a43c-21e8-4446-9759-2f33c634773e_isr_871043_57.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash a5b5ff52b2e8c8b6459a926d1b5323f8 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c01a6f11-cb57-4b19-937d-8e40904e709e_isr_871042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c01a6f11-cb57-4b19-937d-8e40904e709e_isr_871042_20.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash d8d05472edc56566252ea60d672faf6b parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6802908a-12b0-4cef-bdf1-baa31e478a37_isr_871043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6802908a-12b0-4cef-bdf1-baa31e478a37_isr_871043_51.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash bafcc695d7ab0207ef0882672e5cf489 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a55f7de3-91af-4082-b3b9-36803bfa6d91_isr_871042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a55f7de3-91af-4082-b3b9-36803bfa6d91_isr_871042_45.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 7f80f2f14d829ffee596424a8ab0df3f parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/7c8f619d-913c-440d-8a34-1408a428a950_isr_871043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/7c8f619d-913c-440d-8a34-1408a428a950_isr_871043_5.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash e1e986b752be37ce90c740bf41fa023e parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e70d7f1-995c-472c-8e31-806ac6a5187c_isr_871042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e70d7f1-995c-472c-8e31-806ac6a5187c_isr_871042_38.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash cd77c9580acf5dc9f6cea8804951aefd parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a9cfec61-8cdc-408c-b338-82bcc88e9573_isr_871042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a9cfec61-8cdc-408c-b338-82bcc88e9573_isr_871042_4.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash f08a6a6317e6a0880cae9148cff6c7a4 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78a8ad1d-3da5-4b4e-9af8-29fbe5fb1f71_isr_871043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78a8ad1d-3da5-4b4e-9af8-29fbe5fb1f71_isr_871043_27.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 97c0868f11c1409ff11b3740527113c5 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9aaa0724-b59c-44b7-82ed-b879ef201424_isr_871042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9aaa0724-b59c-44b7-82ed-b879ef201424_isr_871042_21.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash cf2f243bff2efb14cf899e9352c20a08 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/89616ef1-3eb5-48bd-b2d1-2fb4f894140a_isr_871042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/89616ef1-3eb5-48bd-b2d1-2fb4f894140a_isr_871042_40.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 1b79f53faa40c04a0c702634e15598a8 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9244a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/fa4a2996-4ba2-4c52-aff9-704dcac2c758_isr_871042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/fa4a2996-4ba2-4c52-aff9-704dcac2c758_isr_871042_54.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash ad8d1217ece84d9303a5f6ba22530827 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9247c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3e88d2a7-4715-47f5-9afa-c747ebeb627c_isr_871041_5.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3e88d2a7-4715-47f5-9afa-c747ebeb627c_isr_871041_5.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 07460f90d480f1125b3606895b7511f6 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3d44c4e3-3ace-4939-9887-75c5a5a81870_isr_871042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3d44c4e3-3ace-4939-9887-75c5a5a81870_isr_871042_24.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 56e00ca27074402d068cfc99a2eabd9a parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b9275f9d-3132-40e2-876f-5d3d34a0f321_isr_871041_31.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b9275f9d-3132-40e2-876f-5d3d34a0f321_isr_871041_31.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 70c7d1f138997c3a17c680e1a808c7ce parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/17ae6056-8ae1-4f40-90f8-ca6c6e2cd85e_isr_871042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/17ae6056-8ae1-4f40-90f8-ca6c6e2cd85e_isr_871042_61.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 5de65522d279232f3ea77c197254cc26 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7bcd3478-f1d2-4f93-8e64-273416673dad_isr_871042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7bcd3478-f1d2-4f93-8e64-273416673dad_isr_871042_50.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash cd2dae4d8f6bc654a1e69676e1c1a5eb parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/24153914-6ad9-46fa-b30e-c9307b52ea8d_isr_871041_36.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/24153914-6ad9-46fa-b30e-c9307b52ea8d_isr_871041_36.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash ff0ab2a9b67947c99c8b415b4c589f1a parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7be79896-061e-43ff-b09c-f2a7546f9a5b_isr_871042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7be79896-061e-43ff-b09c-f2a7546f9a5b_isr_871042_17.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash af65a5668e16a5a0a86b8a9ae92b2d5a parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d4537a00-4064-4f07-899c-198a2dd0120c_isr_871043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d4537a00-4064-4f07-899c-198a2dd0120c_isr_871043_40.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 5b950742665220a897e0af584ee36966 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9260c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7d80e86-8bce-416b-8300-12d929301eeb_isr_871043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7d80e86-8bce-416b-8300-12d929301eeb_isr_871043_11.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 415d40fb1679696d4be8424d3130aef8 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9263e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9282c598-d4c8-4d55-b21a-929dd3848fb0_isr_871043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9282c598-d4c8-4d55-b21a-929dd3848fb0_isr_871043_45.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 4b34f79164b6bbf44b2f0dbf910d1ad9 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f10101eb-a027-47b6-b530-a7336d717c9c_isr_871042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f10101eb-a027-47b6-b530-a7336d717c9c_isr_871042_9.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash b1c7e8b358fc6540911f16ec2316435d parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3a0a3366-93ae-42e9-bfc4-24cf0e82eb3c_isr_871043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3a0a3366-93ae-42e9-bfc4-24cf0e82eb3c_isr_871043_43.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 75cad09695b06061f24f3f56a2b784ba parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c803f21b-658d-4e8d-b99b-d8cb0c4f5e20_isr_871041_44.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c803f21b-658d-4e8d-b99b-d8cb0c4f5e20_isr_871041_44.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 7bab6addef5c5099e2c3e8aac6322082 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/66aae4ba-7598-4b4f-b877-07e01859602f_isr_871042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/66aae4ba-7598-4b4f-b877-07e01859602f_isr_871042_7.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 98e6b441a111174d99b6d60300df1d2d parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/abeb62e1-4adf-4653-9b18-254a98181800_isr_871041_32.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/abeb62e1-4adf-4653-9b18-254a98181800_isr_871041_32.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash bb78165ba0a5b81cb23261070ea6deb3 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9276a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dceba791-8cfa-4fc4-a768-490cf2301477_isr_871042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dceba791-8cfa-4fc4-a768-490cf2301477_isr_871042_35.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 28998c9fe183a3014575855704d980c5 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9279c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/17922236-6769-4697-abe2-8ccdbefb261a_isr_871041_38.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/17922236-6769-4697-abe2-8ccdbefb261a_isr_871041_38.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash d38ebbb0234acdc21014d9d378da31a8 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/30149f5a-0282-4f45-8d97-8fb259ce60a5_isr_871042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/30149f5a-0282-4f45-8d97-8fb259ce60a5_isr_871042_55.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 8f5e99dc1f5e88729c36f7e44ecc4739 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78145b76-605c-4880-8477-b705e69fa6d8_isr_871043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78145b76-605c-4880-8477-b705e69fa6d8_isr_871043_42.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash ca45ab689e51db9bfc1aae4c69c09134 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0b9aefea-0eb9-44ed-b871-89eb28672394_isr_871042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0b9aefea-0eb9-44ed-b871-89eb28672394_isr_871042_26.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 2884b2b22e3a3aea79bd7cd344b49d3d parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/27960f48-a19f-4813-b07c-ef4fe5c43a6f_isr_871041_49.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/27960f48-a19f-4813-b07c-ef4fe5c43a6f_isr_871041_49.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash b5081d73cf5f0cf4843dd6a2c7cdd2fb parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e810016-4ccd-439a-81ab-f62e1e94de40_isr_871042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e810016-4ccd-439a-81ab-f62e1e94de40_isr_871042_48.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 81cb7e38a8c4e32351379cb6d1c585e8 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/23497de9-4bd3-4786-bd1e-1d54927706e1_isr_871041_43.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/23497de9-4bd3-4786-bd1e-1d54927706e1_isr_871041_43.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 7578dddcbcff087b51bab3ac44eb331c parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b83baa86-ce89-407a-9f0f-254ebdb5871f_isr_871041_35.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b83baa86-ce89-407a-9f0f-254ebdb5871f_isr_871041_35.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash bb9ba62af583bb36e277c3bf50700261 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/67bf4cc8-58d3-437b-8f5f-66d8afc48730_isr_871043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/67bf4cc8-58d3-437b-8f5f-66d8afc48730_isr_871043_34.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash b49e17f8118acb01e7973ce9f69b882e parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c68d034d-5c6c-4be2-84a8-dd4f703ae3e7_isr_871041_42.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c68d034d-5c6c-4be2-84a8-dd4f703ae3e7_isr_871041_42.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 52d605682ceb0606996ede342daf1a5c parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e37fbae-f1dc-4cf7-b930-2fb8450acb6a_isr_871042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e37fbae-f1dc-4cf7-b930-2fb8450acb6a_isr_871042_30.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 87337af192350f230f4ea7cbc8733333 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6a531955-1fb1-423e-b11c-41b764934121_isr_871043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6a531955-1fb1-423e-b11c-41b764934121_isr_871043_6.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash b94ae1bdfe12f48b90fa150152b3c157 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e3a5446-fd7e-4cfa-b9f7-4a06fe3a0024_isr_871042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e3a5446-fd7e-4cfa-b9f7-4a06fe3a0024_isr_871042_15.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash b0619eed0dee30b06a770ce72e11b64c parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/44363731-28ea-4675-8621-b8af47a7ff7f_isr_871043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/44363731-28ea-4675-8621-b8af47a7ff7f_isr_871043_9.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash c35d81f5601bfdbd9d58a50bd48cf50d parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e390578c-b8ae-44ae-9cd4-af79900e2da8_isr_871042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e390578c-b8ae-44ae-9cd4-af79900e2da8_isr_871042_3.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 5fed6ffd17c08fa9e108056a5e48d7ed parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/89f8c8fa-1fbf-479b-a303-9c2e8b993659_isr_871043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/89f8c8fa-1fbf-479b-a303-9c2e8b993659_isr_871043_52.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash a8e78e3905744f289c4da223283f6202 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c4ebb6cd-b7bd-4500-838b-50a131df732b_isr_871043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c4ebb6cd-b7bd-4500-838b-50a131df732b_isr_871043_3.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 0366a6a8dc493a3ee29ca919e78d9c8d parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d3bdf060-0e2c-46fb-bb1d-6c1d62d99355_isr_871043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d3bdf060-0e2c-46fb-bb1d-6c1d62d99355_isr_871043_26.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 13868688c2ecfe22975fd49ef0552113 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2063074-0276-480e-8030-32045e7acab9_isr_871042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2063074-0276-480e-8030-32045e7acab9_isr_871042_58.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 3b7c95a43244a79ed2128f51ea7d0852 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/534da3fc-ea2a-49c0-ba5d-14ef88bb7d51_isr_871041_25.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/534da3fc-ea2a-49c0-ba5d-14ef88bb7d51_isr_871041_25.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 25835397f75c2a67d2378b59382f3326 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/28e172bf-cda4-49e2-8b62-ea49ad801473_isr_871042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/28e172bf-cda4-49e2-8b62-ea49ad801473_isr_871042_23.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 15d469adbb3a609dab9efc55c97fc014 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-10 15:23:52 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c68ffa9-8a18-492c-bbab-089577119a18_isr_871041_2.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c68ffa9-8a18-492c-bbab-089577119a18_isr_871041_2.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 825a37c2102db355b0ae651327f00cad parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a8a243fa-52a5-425b-9f3f-eaa224c6f89f_isr_871043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a8a243fa-52a5-425b-9f3f-eaa224c6f89f_isr_871043_61.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash fb320aac630c646832e7907231b4b2e3 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/541d5374-8163-4f19-b18f-a6faa8916ade_isr_871042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/541d5374-8163-4f19-b18f-a6faa8916ade_isr_871042_28.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash be52345890fddd529beadfc333b6b21d parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/240d3311-88fb-4347-8479-76af4eb0e17e_isr_871041_34.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/240d3311-88fb-4347-8479-76af4eb0e17e_isr_871041_34.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 59593a56198f851ea90a5b4fd383e598 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/13f0af7c-2e70-429f-92e7-cb091e8f4ec8_isr_871042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/13f0af7c-2e70-429f-92e7-cb091e8f4ec8_isr_871042_51.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash c2fdb7a31185a597b426f1fa70b71f46 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b9c9983f-c8fc-4156-b260-d5aabe2208e5_isr_871042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b9c9983f-c8fc-4156-b260-d5aabe2208e5_isr_871042_6.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash ff4e9755d25327d252e470e7d4923423 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4dcd2ec4-14c0-46d8-98ae-36c01e1a7d18_isr_871042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4dcd2ec4-14c0-46d8-98ae-36c01e1a7d18_isr_871042_32.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 224b45e6560bfd2d2f5063d34d6da0fc parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/679220e4-3750-4816-b9f0-3fad43bda335_isr_871043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/679220e4-3750-4816-b9f0-3fad43bda335_isr_871043_29.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 0d1d3b1684815adf29e3c728353ea38e parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/47d6bb58-c1f5-4a02-8e40-18ce8f045be1_isr_871043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/47d6bb58-c1f5-4a02-8e40-18ce8f045be1_isr_871043_41.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash c1ce79b58fb54a3984e2a0d6915645f5 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c13709dd-11c1-4cc4-adad-a30f19b64db3_isr_871041_6.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c13709dd-11c1-4cc4-adad-a30f19b64db3_isr_871041_6.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 064ec82dce9b4c0b9494d83e411e4db4 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/05585126-faa4-4907-a361-da2fa67a63d2_isr_871043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/05585126-faa4-4907-a361-da2fa67a63d2_isr_871043_47.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 7b395d3e4e426264f1f46166c3998eaa parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/f03129ea-a793-42ea-9836-1ddc95680487_isr_871043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/f03129ea-a793-42ea-9836-1ddc95680487_isr_871043_10.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 727c82ede8f2cbadf7d35dd7d9263f4e parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/18573897-4338-4df9-b8ed-fed85a7e5c45_isr_871041_4.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/18573897-4338-4df9-b8ed-fed85a7e5c45_isr_871041_4.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 42064ab26305ca0d75dd4092b8dd9566 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7a82cbbb-ccd9-4a7b-b91d-08b7a90e982d_isr_871042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7a82cbbb-ccd9-4a7b-b91d-08b7a90e982d_isr_871042_47.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 41cd4547cbb9c315cded79e3f6815692 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c137ba99-b06d-4779-9a72-63182c3f882c_isr_871042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c137ba99-b06d-4779-9a72-63182c3f882c_isr_871042_46.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 5c4ef386f3417da2c15ee5aa9af035e7 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/56b468bf-7778-408e-b851-f3a04dc987b9_isr_871041_16.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/56b468bf-7778-408e-b851-f3a04dc987b9_isr_871041_16.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 49ee5cafecd6c358b4b998695f8dbd11 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/35be8e8b-2fda-4fc3-94ce-dcea98f15620_isr_871043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/35be8e8b-2fda-4fc3-94ce-dcea98f15620_isr_871043_59.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash a4b992da060f462af85c5d77730e8ad8 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 102 parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/153f692e-fbb6-41b5-89aa-d5712718ce9e_isr_871041_59.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/153f692e-fbb6-41b5-89aa-d5712718ce9e_isr_871041_59.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash ef81f17dbf2e95d5a313bd9e4c3c31ca parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/22496f55-707c-467b-852e-c981f4a5286f_isr_871042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/22496f55-707c-467b-852e-c981f4a5286f_isr_871042_52.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 9ba1a8d5699ee4051b200ffb3e85bb1b parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e3c7694a-97b2-4e54-a54b-acb66d4cc97d_isr_871041_46.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e3c7694a-97b2-4e54-a54b-acb66d4cc97d_isr_871041_46.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash a57ac4107f087ffbde16fe8dcc696ba2 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9b5d2b9-a1bb-48d9-98c2-a7f3ab87f587_isr_871043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9b5d2b9-a1bb-48d9-98c2-a7f3ab87f587_isr_871043_53.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 2762b5486879eda41756eddcdcb632b5 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4d722da-22ec-492b-bfe6-35a2943a92f4_isr_871043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4d722da-22ec-492b-bfe6-35a2943a92f4_isr_871043_39.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash cf1b5bcfeefad6fbf02984ec3f734f09 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Executor multi has 102 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 25 more blocks parsl.executors.status_handling INFO: Scaling out by 25 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-10 15:23:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-10 15:23:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 15:23:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ef89b7d5-6bdb-476c-8c75-d36e442728fa_isr_871041_23.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ef89b7d5-6bdb-476c-8c75-d36e442728fa_isr_871041_23.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 2336ba03de87a11531b57f17f7603402 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e58deb7e-bfc0-42ed-8903-ae6dcaf23d00_isr_871041_28.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e58deb7e-bfc0-42ed-8903-ae6dcaf23d00_isr_871041_28.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash b570e39318edca18370385c14d9ee693 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2191922c-bb64-40ad-b251-0e94c86e1733_isr_871042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2191922c-bb64-40ad-b251-0e94c86e1733_isr_871042_25.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 5b93aefad1d29e44d4d2ad76f55e59ca parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58ad7c14-5d83-425b-ba9e-9b9e0af979b4_isr_871041_48.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58ad7c14-5d83-425b-ba9e-9b9e0af979b4_isr_871041_48.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 2a64b7c0ea14e7b0b8512f2eeaf40a65 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/bb993842-0051-48a9-8d70-813b5f55472f_isr_871043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/bb993842-0051-48a9-8d70-813b5f55472f_isr_871043_25.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 8e28a0412ba91a7ab4424cefda862f74 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/032eab5d-cbaf-4ba4-b908-a24dfe149f8e_isr_871041_60.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/032eab5d-cbaf-4ba4-b908-a24dfe149f8e_isr_871041_60.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 03160ee5bf882351641093e9d2c2ac77 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b51aa6eb-1f6e-494c-a63b-4ed0ca5bea32_isr_871043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b51aa6eb-1f6e-494c-a63b-4ed0ca5bea32_isr_871043_38.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 138abbbde8a8ca3024a50a145cd1a418 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/dac3f521-c4de-4243-9d60-3fb9f52cba84_isr_871041_22.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/dac3f521-c4de-4243-9d60-3fb9f52cba84_isr_871041_22.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 445fb03c10dbfd617fbd3ab09feabec9 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7d3b0471-25bb-4da7-9283-658002ba8f7c_isr_871041_10.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7d3b0471-25bb-4da7-9283-658002ba8f7c_isr_871041_10.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 81a1b6709f01531bb301d5ebc5b2935a parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/31e94397-ffaa-475a-b93f-7ffb6bd18a00_isr_871041_24.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/31e94397-ffaa-475a-b93f-7ffb6bd18a00_isr_871041_24.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash a735168c76983e3e10ff6e3119ea758f parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a233b837-2dbf-49ef-9162-4d35e5b92d42_isr_871041_21.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a233b837-2dbf-49ef-9162-4d35e5b92d42_isr_871041_21.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash cea5136b891939b96ad74d96ed124b5e parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/86932859-34e7-4cc5-a8fc-d6b97d81eeac_isr_871041_52.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/86932859-34e7-4cc5-a8fc-d6b97d81eeac_isr_871041_52.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 9ddf5b24cd10a18e26b81500a59d5d10 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/65dddadb-e29e-49a4-9302-eb5550e74fce_isr_871042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/65dddadb-e29e-49a4-9302-eb5550e74fce_isr_871042_37.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 20beaf8d2a26e23fc75cd40c774ad7b9 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/709e92b7-d1ec-490a-9069-1a9827d78ca1_isr_871043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/709e92b7-d1ec-490a-9069-1a9827d78ca1_isr_871043_62.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash e2105a5010e471f102968454a6c8f8b5 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4b44517-8526-481c-a6e0-02010b9c5421_isr_871043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4b44517-8526-481c-a6e0-02010b9c5421_isr_871043_54.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 46f52306377377b641d3908874b0b01d parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bc6e56e0-1fd3-44f3-b3ef-4deb567ad593_isr_871041_8.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bc6e56e0-1fd3-44f3-b3ef-4deb567ad593_isr_871041_8.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash d0bc304ffa69e991db8ec29341f2a275 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7baf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/18ffb3cb-4758-482c-8091-256e32d64c07_isr_871042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/18ffb3cb-4758-482c-8091-256e32d64c07_isr_871042_60.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 7d8a2790bd8dff58b2eb72998baeb1c5 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a2ca3c9b-de23-426b-b84c-5dc904cfa1f1_isr_871041_12.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a2ca3c9b-de23-426b-b84c-5dc904cfa1f1_isr_871041_12.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 64418e77ef583d67b7f9eeb5b0a95c35 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c8485cd-c62f-49cc-b26b-22d9a609cefd_isr_871041_37.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c8485cd-c62f-49cc-b26b-22d9a609cefd_isr_871041_37.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 1ee658cba5bbbd195f0f8619ed7e715a parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2e8bd4d7-e8b5-44d9-91f4-657281bb9ea5_isr_871041_29.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2e8bd4d7-e8b5-44d9-91f4-657281bb9ea5_isr_871041_29.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash b3e09cd40928c80715c09cd4a77d1c10 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3c67408f-304e-4703-bb42-000b3b08a561_isr_871041_9.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3c67408f-304e-4703-bb42-000b3b08a561_isr_871041_9.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash e39ad29f2f8ed1a4c4c305ceccc07b71 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ae93778e-37cf-4ad9-bff8-d4d62f088ea9_isr_871041_20.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ae93778e-37cf-4ad9-bff8-d4d62f088ea9_isr_871041_20.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash b510c6255e5b27a1045026254911ef40 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7729e41a-3d40-481d-b224-9f9217d1649a_isr_871041_61.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7729e41a-3d40-481d-b224-9f9217d1649a_isr_871041_61.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 0ab068a5f51b1671626004616d7690fb parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/464fef7c-3c86-412a-9e6a-7a2bacc47c41_isr_871041_27.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/464fef7c-3c86-412a-9e6a-7a2bacc47c41_isr_871041_27.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash d49a27396c35a2e8f5416ee1eb32fe00 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/ad180198-00b7-4b80-b6c5-063c8edcf97a_isr_871042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/ad180198-00b7-4b80-b6c5-063c8edcf97a_isr_871042_44.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 0c85fd9b71a869ef270f308629b66761 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/042d4612-b175-47c6-a736-39e45e4adf8d_isr_871043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/042d4612-b175-47c6-a736-39e45e4adf8d_isr_871043_16.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 753aa46626412d702121c8fe9321c953 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/942d2471-8be4-47ee-ae83-77b2cee36e3d_isr_871042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/942d2471-8be4-47ee-ae83-77b2cee36e3d_isr_871042_22.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 7e1823bcae93db95e75f3e7248bcd05d parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b17c2b1c-17e1-4d40-8450-d225d4a75bb5_isr_871042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b17c2b1c-17e1-4d40-8450-d225d4a75bb5_isr_871042_10.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash d2ff7cbe195e68e359dfb9813cc77341 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e3862ae-e909-422c-ad38-d15e52b22474_isr_871043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e3862ae-e909-422c-ad38-d15e52b22474_isr_871043_18.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash c6f9bda40b57bb993a6a033866915231 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/682493d4-fb8f-4ffb-bd1f-aa83204535f5_isr_871043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/682493d4-fb8f-4ffb-bd1f-aa83204535f5_isr_871043_55.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 4bc19635ebd5150d31f6a7bccd1fa121 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d24c4c77-def9-4592-b8ab-f08b89db0123_isr_871043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d24c4c77-def9-4592-b8ab-f08b89db0123_isr_871043_17.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 12e9935a7a28e0d8b110b8e8455e4fd3 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07fb4eab-4f65-466b-853a-e09f2db9eaa0_isr_871042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07fb4eab-4f65-466b-853a-e09f2db9eaa0_isr_871042_19.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash d12cff384d64b03bb3eaf7a7a6e0efc7 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7a16f9d-2543-4438-8119-a924cd216c0e_isr_871043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7a16f9d-2543-4438-8119-a924cd216c0e_isr_871043_12.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash df028053cddaaa75fdff34ed19be6c3d parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f135a6af-18a1-4884-8a9d-0cfec8f361ba_isr_871042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f135a6af-18a1-4884-8a9d-0cfec8f361ba_isr_871042_36.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash cb148d1c509d7f5ac3442fd4938e1688 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7faf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2ef82e96-a5da-48cb-a93e-976c5e2bf7be_isr_871041_7.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2ef82e96-a5da-48cb-a93e-976c5e2bf7be_isr_871041_7.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 4c49e4391d96f5688270ae72ad468b24 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7c5bd345-aaeb-4cd1-b77b-3b3364616f82_isr_871041_58.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7c5bd345-aaeb-4cd1-b77b-3b3364616f82_isr_871041_58.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 65df6bed839cf6847614b2287b456cb8 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a29002f2-d78e-4902-bb9b-82d594dd15fb_isr_871043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a29002f2-d78e-4902-bb9b-82d594dd15fb_isr_871043_60.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 6e5e3a1ed9dfec3db790c63666a6add9 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07e93b91-7d58-4095-b791-6cb78ac3a193_isr_871042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07e93b91-7d58-4095-b791-6cb78ac3a193_isr_871042_18.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash da24ba1d166a4173bef8aaabc03f8c03 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/eabe1f84-7b7a-4e4a-8c33-ac54da871bb9_isr_871041_14.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/eabe1f84-7b7a-4e4a-8c33-ac54da871bb9_isr_871041_14.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 7e421c277ef8ef6434e1294031584ae6 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ce179a3e-1809-4243-ace3-4b2e78469c0a_isr_871043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ce179a3e-1809-4243-ace3-4b2e78469c0a_isr_871043_14.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 3a71b13fba162721bbffaab8ffa9a336 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3bb7eda1-7d3b-4e21-95c3-5ee8a7aded50_isr_871042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3bb7eda1-7d3b-4e21-95c3-5ee8a7aded50_isr_871042_1.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 3b1d06230839f7b36fd99fd2e687c265 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ad1cfbd0-0892-41cf-aa55-bc4cbcfa8fcf_isr_871043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ad1cfbd0-0892-41cf-aa55-bc4cbcfa8fcf_isr_871043_46.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 71ff5dd16be988f9c1c3c69d80c02891 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2a1a3598-6e85-44cd-8fab-0723eb036801_isr_871042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2a1a3598-6e85-44cd-8fab-0723eb036801_isr_871042_29.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 337ca1b988a337004249793c68a0107d parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/82a49231-2042-47ad-b96d-553768e160f5_isr_871041_39.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/82a49231-2042-47ad-b96d-553768e160f5_isr_871041_39.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 9316bcb30b56729c0afe3d15a6b8c2d9 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/963c6033-b710-4a09-b4b1-ca018e8f3e1b_isr_871043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/963c6033-b710-4a09-b4b1-ca018e8f3e1b_isr_871043_23.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 56d3972f47cc8a224c4d334fbb3cb9ee parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8411c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0bc39768-a801-4576-8fc4-fc581f59f30b_isr_871041_11.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0bc39768-a801-4576-8fc4-fc581f59f30b_isr_871041_11.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 582af961332db966351e98fa68620028 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3179cf69-86e4-42c5-aa27-2e8d98f7b893_isr_871043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3179cf69-86e4-42c5-aa27-2e8d98f7b893_isr_871043_31.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash d43019cba7f9e5142e82203aff9bd610 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58e404b2-3878-4abd-acec-7970568638cb_isr_871041_45.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58e404b2-3878-4abd-acec-7970568638cb_isr_871041_45.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 84288f2cf27167a3befdba9eff691e90 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/9fc7fbb0-1d04-4221-9d56-9c8aae52059b_isr_871041_50.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/9fc7fbb0-1d04-4221-9d56-9c8aae52059b_isr_871041_50.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash f3e7e408fe16a66aa4008d8fb81047a1 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6b0d3ff7-0d6e-4de3-949d-a3bd8d028f99_isr_871041_47.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6b0d3ff7-0d6e-4de3-949d-a3bd8d028f99_isr_871041_47.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 5cfc7364c3323f1db6092f67d37ec4ed parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/01de7f3c-b47c-4994-a7bf-ec13a5ad0233_isr_871043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/01de7f3c-b47c-4994-a7bf-ec13a5ad0233_isr_871043_19.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash e4ae0cab1f78dc53d7a8a82491be8055 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/faa9f722-3437-462d-86e8-971f6e866533_isr_871042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/faa9f722-3437-462d-86e8-971f6e866533_isr_871042_2.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 9118327a6fb2649c9174833e9d91cba4 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8427a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0124d4ed-346c-4610-acfc-989fc5961659_isr_871041_1.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0124d4ed-346c-4610-acfc-989fc5961659_isr_871041_1.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash b3a4747dafb3fa455fece4eb80d0ae72 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1dfb427f-ad94-4abd-945f-786681a77102_isr_871042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1dfb427f-ad94-4abd-945f-786681a77102_isr_871042_49.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 845b1f84163cd90b4c304a35ff415104 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/82686b52-1297-4e40-9569-626dab7b7cd7_isr_871042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/82686b52-1297-4e40-9569-626dab7b7cd7_isr_871042_14.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash ba0c1a0eaa012a410e61bed36ee66d68 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2da3e8a4-023c-4eac-86b0-dd611a06a994_isr_871042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2da3e8a4-023c-4eac-86b0-dd611a06a994_isr_871042_43.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 54219c02b0302d3b0a57ca65eb8e978a parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-10 15:23:53 proc_lsst.multi:170 [INFO] [multi] job_id 16980029 proc_lsst.multi INFO: [multi] job_id 16980029 2024-03-10 15:23:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980029 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980029 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16980029 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-10 15:23:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-10 15:23:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf95ca21-df4f-43b7-9523-40b7030f991a_isr_871041_62.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf95ca21-df4f-43b7-9523-40b7030f991a_isr_871041_62.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash b0097b7ad94fd2b931caa8d195ecb655 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8434c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bb1a6adb-f72c-4b2d-9c26-216afd34c56f_isr_871041_41.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bb1a6adb-f72c-4b2d-9c26-216afd34c56f_isr_871041_41.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash b346687cb06caa5696a6eeb368362b42 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9887a2c9-73ee-4ed1-b8c6-f40454b1f063_isr_871043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9887a2c9-73ee-4ed1-b8c6-f40454b1f063_isr_871043_32.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 40a2355b5eb8e326e59499acda6ed5ef parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/35e9b0a8-3ad4-4e21-a11d-75a12f4c3e0f_isr_871042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/35e9b0a8-3ad4-4e21-a11d-75a12f4c3e0f_isr_871042_41.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash b03da42d0f7bc33405bc41623f4c51a9 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ec10aca4-12f6-4a90-8c8a-04cf1158ca6d_isr_871041_40.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ec10aca4-12f6-4a90-8c8a-04cf1158ca6d_isr_871041_40.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 7f3786614cb45cf2811e645c7556680e parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dabcad65-3d80-44bb-91a0-2de160008121_isr_871043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dabcad65-3d80-44bb-91a0-2de160008121_isr_871043_44.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash ecb069c0bc2128ae20e901661cc18552 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9ac5a7c-510c-4f5f-aff2-9740b1c09c20_isr_871043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9ac5a7c-510c-4f5f-aff2-9740b1c09c20_isr_871043_36.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash a743efcec1da381cae0a4dd7f75a14c5 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/66593e27-83d5-473f-9e32-b3ab8b1df53c_isr_871043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/66593e27-83d5-473f-9e32-b3ab8b1df53c_isr_871043_24.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 35b65a4e0ce30e695bafa68aafa3ba1b parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/808ef37f-1c3f-448d-85f4-622d201eeacd_isr_871043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/808ef37f-1c3f-448d-85f4-622d201eeacd_isr_871043_28.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash cd63cf6ce7791f0584c8c08f23e876fe parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/30192b0b-29ce-4c11-9fe8-e3790ba0aed9_isr_871041_51.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/30192b0b-29ce-4c11-9fe8-e3790ba0aed9_isr_871041_51.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 61d5994a4ae60414dd81275203960eb8 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b95a9700-0bf4-4334-ae93-757d23f255ad_isr_871043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b95a9700-0bf4-4334-ae93-757d23f255ad_isr_871043_35.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash db87698e8a66960d6a41edeb5bea322f parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/797fecb2-292f-4c1a-9e1b-abae4350368e_isr_871043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/797fecb2-292f-4c1a-9e1b-abae4350368e_isr_871043_8.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 7c696607f8840dae2652b087f38d0a76 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/15e707c0-0252-43c2-a1d7-217de83ce96c_isr_871043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/15e707c0-0252-43c2-a1d7-217de83ce96c_isr_871043_1.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash c84ef4f86d0e880ff05c06ec900a0cf3 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/12f53ada-c75d-4040-b66f-76f6eba68585_isr_871043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/12f53ada-c75d-4040-b66f-76f6eba68585_isr_871043_33.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 3745dedcb73719de9bca8bcc05e29412 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/883278ec-1a3a-47a1-b8ac-e15bb6ccea71_isr_871041_15.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/883278ec-1a3a-47a1-b8ac-e15bb6ccea71_isr_871041_15.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 819f0ff423e4cec340f9d2c9c1a25070 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6ff582d9-cdb0-4cce-a02f-992cbe918914_isr_871041_17.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6ff582d9-cdb0-4cce-a02f-992cbe918914_isr_871041_17.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash a99a43d23edbf2f5cf8753c83db5041f parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/8abceed8-80ed-4b4b-b05d-36507515e531_isr_871043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/8abceed8-80ed-4b4b-b05d-36507515e531_isr_871043_56.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 77d95e2b30694dba8e33a887844cda79 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e618bac0-7a9e-4017-9415-9f221a31c6e2_isr_871042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e618bac0-7a9e-4017-9415-9f221a31c6e2_isr_871042_34.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 8841f85630e2e1576076c2bd537df7d4 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/493021eb-fb97-4625-9c12-6f306cf3569f_isr_871043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/493021eb-fb97-4625-9c12-6f306cf3569f_isr_871043_7.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash ae48e6b4992034c49c1c0cca0cf533f8 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6c7a413f-a1f3-42c1-841d-cdae62a449bb_isr_871043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6c7a413f-a1f3-42c1-841d-cdae62a449bb_isr_871043_20.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash b6032fed405fa930a70e52e007143576 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/cd96324c-f7a1-4837-a672-a52e85ca6e0a_isr_871042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/cd96324c-f7a1-4837-a672-a52e85ca6e0a_isr_871042_39.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping 2024-03-10 15:23:53 proc_lsst.multi:170 [INFO] [multi] job_id 16980030 proc_lsst.multi INFO: [multi] job_id 16980030 2024-03-10 15:23:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 15:23:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980030 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980030 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16980030 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-10 15:23:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-10 15:23:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 15:23:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:54 proc_lsst.multi:170 [INFO] [multi] job_id 16980031 proc_lsst.multi INFO: [multi] job_id 16980031 2024-03-10 15:23:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 15:23:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980031 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980031 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16980031 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-10 15:23:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-10 15:23:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 15:23:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:54 proc_lsst.multi:170 [INFO] [multi] job_id 16980032 proc_lsst.multi INFO: [multi] job_id 16980032 2024-03-10 15:23:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 15:23:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980032 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980032 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16980032 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-10 15:23:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-10 15:23:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 15:23:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:54 proc_lsst.multi:170 [INFO] [multi] job_id 16980033 proc_lsst.multi INFO: [multi] job_id 16980033 2024-03-10 15:23:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:23:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980033 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980033 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16980033 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-10 15:23:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-10 15:23:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:23:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:55 proc_lsst.multi:170 [INFO] [multi] job_id 16980034 proc_lsst.multi INFO: [multi] job_id 16980034 2024-03-10 15:23:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 15:23:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980034 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980034 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16980034 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-10 15:23:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-10 15:23:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 15:23:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:55 proc_lsst.multi:170 [INFO] [multi] job_id 16980035 proc_lsst.multi INFO: [multi] job_id 16980035 2024-03-10 15:23:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 15:23:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980035 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980035 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16980035 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-10 15:23:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-10 15:23:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 15:23:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:56 proc_lsst.multi:170 [INFO] [multi] job_id 16980036 proc_lsst.multi INFO: [multi] job_id 16980036 2024-03-10 15:23:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 15:23:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980036 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980036 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16980036 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-10 15:23:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-10 15:23:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 15:23:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:56 proc_lsst.multi:170 [INFO] [multi] job_id 16980037 proc_lsst.multi INFO: [multi] job_id 16980037 2024-03-10 15:23:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 15:23:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980037 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980037 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16980037 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-10 15:23:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-10 15:23:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 15:23:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:57 proc_lsst.multi:170 [INFO] [multi] job_id 16980038 proc_lsst.multi INFO: [multi] job_id 16980038 2024-03-10 15:23:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 15:23:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980038 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980038 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16980038 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-10 15:23:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-10 15:23:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 15:23:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:57 proc_lsst.multi:170 [INFO] [multi] job_id 16980039 proc_lsst.multi INFO: [multi] job_id 16980039 2024-03-10 15:23:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-10 15:23:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980039 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980039 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16980039 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-10 15:23:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-10 15:23:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-10 15:23:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:58 proc_lsst.multi:170 [INFO] [multi] job_id 16980040 proc_lsst.multi INFO: [multi] job_id 16980040 2024-03-10 15:23:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-10 15:23:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980040 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980040 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16980040 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-10 15:23:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-10 15:23:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-10 15:23:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:58 proc_lsst.multi:170 [INFO] [multi] job_id 16980041 proc_lsst.multi INFO: [multi] job_id 16980041 2024-03-10 15:23:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-10 15:23:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980041 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980041 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16980041 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-10 15:23:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-10 15:23:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-10 15:23:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:59 proc_lsst.multi:170 [INFO] [multi] job_id 16980042 proc_lsst.multi INFO: [multi] job_id 16980042 2024-03-10 15:23:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-10 15:23:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980042 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980042 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16980042 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-10 15:23:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-10 15:23:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-10 15:23:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:59 proc_lsst.multi:170 [INFO] [multi] job_id 16980043 proc_lsst.multi INFO: [multi] job_id 16980043 2024-03-10 15:23:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-10 15:23:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980043 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980043 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16980043 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-10 15:23:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-10 15:23:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-10 15:23:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:23:59 proc_lsst.multi:170 [INFO] [multi] job_id 16980044 proc_lsst.multi INFO: [multi] job_id 16980044 2024-03-10 15:23:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-10 15:23:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980044 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980044 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16980044 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-10 15:23:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-10 15:23:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:23:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:23:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:23:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:23:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-10 15:23:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:23:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:00 proc_lsst.multi:170 [INFO] [multi] job_id 16980045 proc_lsst.multi INFO: [multi] job_id 16980045 2024-03-10 15:24:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-10 15:24:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980045 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980045 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16980045 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-10 15:24:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-10 15:24:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-10 15:24:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:01 proc_lsst.multi:170 [INFO] [multi] job_id 16980046 proc_lsst.multi INFO: [multi] job_id 16980046 2024-03-10 15:24:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-10 15:24:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980046 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980046 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16980046 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-10 15:24:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-10 15:24:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-10 15:24:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:01 proc_lsst.multi:170 [INFO] [multi] job_id 16980047 proc_lsst.multi INFO: [multi] job_id 16980047 2024-03-10 15:24:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-10 15:24:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980047 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980047 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16980047 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-10 15:24:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-10 15:24:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-10 15:24:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:01 proc_lsst.multi:170 [INFO] [multi] job_id 16980048 proc_lsst.multi INFO: [multi] job_id 16980048 2024-03-10 15:24:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-10 15:24:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980048 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980048 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16980048 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-10 15:24:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-10 15:24:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-10 15:24:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:02 proc_lsst.multi:170 [INFO] [multi] job_id 16980049 proc_lsst.multi INFO: [multi] job_id 16980049 2024-03-10 15:24:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-10 15:24:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980049 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980049 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16980049 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-10 15:24:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-10 15:24:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-10 15:24:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:02 proc_lsst.multi:170 [INFO] [multi] job_id 16980050 proc_lsst.multi INFO: [multi] job_id 16980050 2024-03-10 15:24:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-10 15:24:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980050 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980050 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16980050 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-10 15:24:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-10 15:24:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-10 15:24:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:03 proc_lsst.multi:170 [INFO] [multi] job_id 16980051 proc_lsst.multi INFO: [multi] job_id 16980051 2024-03-10 15:24:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-10 15:24:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980051 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980051 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16980051 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-10 15:24:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-10 15:24:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-10 15:24:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:03 proc_lsst.multi:170 [INFO] [multi] job_id 16980052 proc_lsst.multi INFO: [multi] job_id 16980052 2024-03-10 15:24:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-10 15:24:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980052 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980052 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16980052 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-10 15:24:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-10 15:24:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-10 15:24:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:04 proc_lsst.multi:170 [INFO] [multi] job_id 16980053 proc_lsst.multi INFO: [multi] job_id 16980053 2024-03-10 15:24:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-10 15:24:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980053 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980053 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16980053 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/25 running/pending blocks, and 5 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 26 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 21 more blocks parsl.executors.status_handling INFO: Scaling out by 21 blocks parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-10 15:24:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-10 15:24:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-10 15:24:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:04 proc_lsst.multi:170 [INFO] [multi] job_id 16980054 proc_lsst.multi INFO: [multi] job_id 16980054 2024-03-10 15:24:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-10 15:24:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980054 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980054 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16980054 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-10 15:24:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-10 15:24:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-10 15:24:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:04 proc_lsst.multi:170 [INFO] [multi] job_id 16980055 proc_lsst.multi INFO: [multi] job_id 16980055 2024-03-10 15:24:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-10 15:24:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980055 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980055 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16980055 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-10 15:24:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-10 15:24:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-10 15:24:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:05 proc_lsst.multi:170 [INFO] [multi] job_id 16980056 proc_lsst.multi INFO: [multi] job_id 16980056 2024-03-10 15:24:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-10 15:24:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980056 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980056 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16980056 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-10 15:24:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-10 15:24:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-10 15:24:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:06 proc_lsst.multi:170 [INFO] [multi] job_id 16980057 proc_lsst.multi INFO: [multi] job_id 16980057 2024-03-10 15:24:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-10 15:24:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980057 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980057 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16980057 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-10 15:24:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:06 proc_lsst.multi:170 [INFO] [multi] job_id 16980058 proc_lsst.multi INFO: [multi] job_id 16980058 2024-03-10 15:24:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16980058 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16980058 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16980058 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-10 15:24:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:06 proc_lsst.multi:170 [INFO] [multi] job_id 16980059 proc_lsst.multi INFO: [multi] job_id 16980059 2024-03-10 15:24:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:06 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16980059 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16980059 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16980059 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-10 15:24:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:07 proc_lsst.multi:170 [INFO] [multi] job_id 16980060 proc_lsst.multi INFO: [multi] job_id 16980060 2024-03-10 15:24:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 15:24:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:07 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16980060 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16980060 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16980060 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-10 15:24:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-10 15:24:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 15:24:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:07 proc_lsst.multi:170 [INFO] [multi] job_id 16980061 proc_lsst.multi INFO: [multi] job_id 16980061 2024-03-10 15:24:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 15:24:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:07 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16980061 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16980061 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16980061 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-10 15:24:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-10 15:24:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 15:24:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:08 proc_lsst.multi:170 [INFO] [multi] job_id 16980062 proc_lsst.multi INFO: [multi] job_id 16980062 2024-03-10 15:24:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 15:24:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:08 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16980062 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16980062 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16980062 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-10 15:24:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-10 15:24:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 15:24:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:09 proc_lsst.multi:170 [INFO] [multi] job_id 16980069 proc_lsst.multi INFO: [multi] job_id 16980069 2024-03-10 15:24:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:09 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16980069 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16980069 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16980069 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-10 15:24:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:09 proc_lsst.multi:170 [INFO] [multi] job_id 16980071 proc_lsst.multi INFO: [multi] job_id 16980071 2024-03-10 15:24:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:09 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980071 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980071 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16980071 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-10 15:24:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:10 proc_lsst.multi:170 [INFO] [multi] job_id 16980072 proc_lsst.multi INFO: [multi] job_id 16980072 2024-03-10 15:24:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 15:24:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:10 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980072 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980072 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16980072 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-10 15:24:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:10 proc_lsst.multi:170 [INFO] [multi] job_id 16980073 proc_lsst.multi INFO: [multi] job_id 16980073 2024-03-10 15:24:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 15:24:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:10 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980073 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980073 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16980073 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-10 15:24:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 15:24:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:11 proc_lsst.multi:170 [INFO] [multi] job_id 16980074 proc_lsst.multi INFO: [multi] job_id 16980074 2024-03-10 15:24:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 15:24:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980074 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980074 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16980074 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-10 15:24:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:11 proc_lsst.multi:170 [INFO] [multi] job_id 16980075 proc_lsst.multi INFO: [multi] job_id 16980075 2024-03-10 15:24:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980075 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980075 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16980075 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-10 15:24:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:12 proc_lsst.multi:170 [INFO] [multi] job_id 16980076 proc_lsst.multi INFO: [multi] job_id 16980076 2024-03-10 15:24:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 15:24:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980076 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980076 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16980076 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-10 15:24:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a180f3a1805e10b0314333669c3f8942 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 67e3719ce10ceca0b32617113873c516 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8be5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/932df572-b445-4842-8f57-0d943efdb51b_characterizeImage_871042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/932df572-b445-4842-8f57-0d943efdb51b_characterizeImage_871042_5.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/8a27ee53-5259-472c-b5dc-0e40c610167b_isr_871042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/8a27ee53-5259-472c-b5dc-0e40c610167b_isr_871042_5.stderr 2024-03-10 15:24:12 proc_lsst.multi:170 [INFO] [multi] job_id 16980077 proc_lsst.multi INFO: [multi] job_id 16980077 2024-03-10 15:24:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 15:24:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980077 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980077 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16980077 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-10 15:24:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:12 proc_lsst.multi:170 [INFO] [multi] job_id 16980078 proc_lsst.multi INFO: [multi] job_id 16980078 2024-03-10 15:24:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 15:24:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980078 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980078 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16980078 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-10 15:24:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 15:24:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:13 proc_lsst.multi:170 [INFO] [multi] job_id 16980079 proc_lsst.multi INFO: [multi] job_id 16980079 2024-03-10 15:24:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 15:24:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980079 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980079 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16980079 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-10 15:24:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b71d270bd82b51860034c525141f4233 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 2c032068987cbe0561a27564aeaf28ab parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8beac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2f93ca81-9302-493d-ac1d-e0ee010cf8d8_characterizeImage_871041_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2f93ca81-9302-493d-ac1d-e0ee010cf8d8_characterizeImage_871041_53.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c25dae18-a6eb-4420-8dfa-f2b3b60ed6cf_isr_871041_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c25dae18-a6eb-4420-8dfa-f2b3b60ed6cf_isr_871041_53.stderr 2024-03-10 15:24:13 proc_lsst.multi:170 [INFO] [multi] job_id 16980080 proc_lsst.multi INFO: [multi] job_id 16980080 2024-03-10 15:24:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 15:24:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980080 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980080 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16980080 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-10 15:24:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 15:24:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 15:24:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 15:24:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,n3010,198.48.92.26,10.64.129.10,127.0.0.1,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54889 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 15:24:14 proc_lsst.multi:170 [INFO] [multi] job_id 16980081 proc_lsst.multi INFO: [multi] job_id 16980081 2024-03-10 15:24:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-10 15:24:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 15:24:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16980081 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16980081 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16980081 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/46 running/pending blocks, and 25 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9191463489bcf1af8624fdd0622630b7 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 9efe3c94bcf0d70098e6b795eb247444 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8befc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6dd65d2f-063f-45ef-a4dd-176e09d37816_characterizeImage_871042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6dd65d2f-063f-45ef-a4dd-176e09d37816_characterizeImage_871042_56.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d48ee22-2fb6-4c8c-a384-306fb442538d_isr_871042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d48ee22-2fb6-4c8c-a384-306fb442538d_isr_871042_56.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/46 running/pending blocks, and 26 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd29caf3d743e400a80e56e1936fd225 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash f2ae214773aad8dd10cd776b3005d9f7 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/96a8a182-5b74-4732-82ad-76c8aeb72e54_isr_871043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/96a8a182-5b74-4732-82ad-76c8aeb72e54_isr_871043_2.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b614491c565d508723c63433e8aa779 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 15129c944d4e00e319e8c55aaa444397 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bcc30fd7-859b-4295-945e-89a2df2171e2_characterizeImage_871041_33.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bcc30fd7-859b-4295-945e-89a2df2171e2_characterizeImage_871041_33.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/db32f59b-076d-49c6-961e-5392169299a3_isr_871041_33.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/db32f59b-076d-49c6-961e-5392169299a3_isr_871041_33.stderr 2024-03-10 15:24:24 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:24:24 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bab9443b0f1259a00e7190ecfb4b9497 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 7184c3e07fa437ad2f778bc8634fc101 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d0e8b56a-9073-4511-ae97-f9626c4957ec_characterizeImage_871042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d0e8b56a-9073-4511-ae97-f9626c4957ec_characterizeImage_871042_11.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4d8057da-dc69-42a8-9340-ff0b27659b58_isr_871042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4d8057da-dc69-42a8-9340-ff0b27659b58_isr_871042_11.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c66e0a5ee9f55e25170a8330f210a0c7 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash aec79722a46162d629833f212753f7cf parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cdb94df5-74de-4f1c-ae92-194c56f9a7de_characterizeImage_871042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cdb94df5-74de-4f1c-ae92-194c56f9a7de_characterizeImage_871042_27.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1bcc3775-1038-4232-ada5-24d6b1b2d71f_isr_871042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1bcc3775-1038-4232-ada5-24d6b1b2d71f_isr_871042_27.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38bb98bc0aa3c9f8a4ee1e366a4e14c6 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 509ccf68978adc63ee1d8ac594dc954c parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7faddab8-3590-4f94-8ab9-6facc6e139e0_characterizeImage_871042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7faddab8-3590-4f94-8ab9-6facc6e139e0_characterizeImage_871042_53.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/bd18eca1-4347-4eaf-a3c6-655316112332_isr_871042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/bd18eca1-4347-4eaf-a3c6-655316112332_isr_871042_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:32 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:33 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:34 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7745f9ff40887c645b654f530238a9cf with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash ee105ba3a8b11ff8c619495f7b34b08e parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9cbedd42-8dcf-4cd1-ad9f-439e55fdf08e_characterizeImage_871042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9cbedd42-8dcf-4cd1-ad9f-439e55fdf08e_characterizeImage_871042_8.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9a003aa3-8ca4-46ae-83db-977823e9caef_isr_871042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9a003aa3-8ca4-46ae-83db-977823e9caef_isr_871042_8.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bddae7a7563ee362f6d85b2ecb3dcee with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 4d4bf75d4d0aadaaa9242ce38d458517 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1d9d56f8-a73f-4885-ac46-619095ea8643_characterizeImage_871042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1d9d56f8-a73f-4885-ac46-619095ea8643_characterizeImage_871042_57.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/777caa82-45bb-4d1a-8cc3-97c7d5e197a2_isr_871042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/777caa82-45bb-4d1a-8cc3-97c7d5e197a2_isr_871042_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:34 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:35 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e5bf69c322cf3477817a12254cee36e with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash d5063454d9460b32ed243f67b345b36c parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5ea73c70-4c2d-4b08-85e5-9c04d4a58826_characterizeImage_871041_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5ea73c70-4c2d-4b08-85e5-9c04d4a58826_characterizeImage_871041_26.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf8a4251-3f7a-4325-9eb9-b2dee2a4472d_isr_871041_26.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf8a4251-3f7a-4325-9eb9-b2dee2a4472d_isr_871041_26.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8de7e41910e98e8f689a012183617763 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash f9d4564fb3a3ef3a4b04c08c4b87469c parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ef03b752-f9bd-4e1c-9953-b740fa34f691_characterizeImage_871041_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ef03b752-f9bd-4e1c-9953-b740fa34f691_characterizeImage_871041_18.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/768e0228-380c-4c92-adee-6a6a541933e5_isr_871041_18.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/768e0228-380c-4c92-adee-6a6a541933e5_isr_871041_18.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4e2d242a115d78bbe3d897f8cbb1288 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 4f3f5f5d593d59ae76fe99f20119df27 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5e6605f9-f0d7-4b6a-b0f8-d2c20e12c99a_characterizeImage_871041_30.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5e6605f9-f0d7-4b6a-b0f8-d2c20e12c99a_characterizeImage_871041_30.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b6450d27-31b2-4450-9520-80117de54def_isr_871041_30.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b6450d27-31b2-4450-9520-80117de54def_isr_871041_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:35 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c48e4a460f2917aaa55fd4fc3132ff3 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash a5c811148fbd580b3ca9d22c12be5dc1 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/cc549fc5-960b-4415-8f3f-947a6cb521b9_characterizeImage_871043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/cc549fc5-960b-4415-8f3f-947a6cb521b9_characterizeImage_871043_21.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/fb96685d-5500-4826-836b-1856f61eeec2_isr_871043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/fb96685d-5500-4826-836b-1856f61eeec2_isr_871043_21.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 486c7a956206e4c89493184bbbd0c711 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 485ce8a3cac4c65599c99090d6af05d9 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/73c43a42-039c-46b8-9068-f6d21225a2f3_characterizeImage_871041_57.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/73c43a42-039c-46b8-9068-f6d21225a2f3_characterizeImage_871041_57.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/936b6d74-ba49-492b-9557-95ee4cd0ef07_isr_871041_57.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/936b6d74-ba49-492b-9557-95ee4cd0ef07_isr_871041_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:36 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f6923037126d354ca68b250da12ca3 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 2b6fe90ab3a1ab2b4b1ee44d9702cd05 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7afd8e16-fd81-4a8b-afbd-d9f995e7e68a_characterizeImage_871041_3.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7afd8e16-fd81-4a8b-afbd-d9f995e7e68a_characterizeImage_871041_3.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/1d7626b1-8278-499f-a7cf-e698f2b0bbc6_isr_871041_3.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/1d7626b1-8278-499f-a7cf-e698f2b0bbc6_isr_871041_3.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ebec140d954ee8a4d8b5a3c0e86efde with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash d32f50f164c0cba68f01765fe2d67cc8 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/24c456c3-5291-4acf-9a89-c819c3c8ece9_characterizeImage_871042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/24c456c3-5291-4acf-9a89-c819c3c8ece9_characterizeImage_871042_31.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/6a1bb29e-a896-49c2-b8e7-e9cfa01ffa23_isr_871042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/6a1bb29e-a896-49c2-b8e7-e9cfa01ffa23_isr_871042_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:36 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c66bc8f808cebdda7d576f67f424722e with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash f536eb641e71ec62a72698dd92eb8197 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/92caf0bc-fc2e-4592-a894-85c3eec994fe_characterizeImage_871042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/92caf0bc-fc2e-4592-a894-85c3eec994fe_characterizeImage_871042_12.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d2e389a-9ef2-4675-994d-f4c45a32e4f7_isr_871042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/5d2e389a-9ef2-4675-994d-f4c45a32e4f7_isr_871042_12.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b16ad4033c7ac9b74e403b69435c2cbb with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash d3d3969e1ba59672a941bded44f94e9d parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7744b24c-9095-4bac-935e-fa9e7ba3da6a_characterizeImage_871041_13.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7744b24c-9095-4bac-935e-fa9e7ba3da6a_characterizeImage_871041_13.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e4b0da5d-a8e0-4c26-a978-bcdf0c644ebf_isr_871041_13.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e4b0da5d-a8e0-4c26-a978-bcdf0c644ebf_isr_871041_13.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baeb27904080355cbd4795640c6f1817 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 3b775a3ea363a3ff67e5f721173967fa parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ea87e91e-6ca4-4fec-bd8f-52bc0bb1a427_characterizeImage_871041_56.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ea87e91e-6ca4-4fec-bd8f-52bc0bb1a427_characterizeImage_871041_56.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c6393991-0ddb-4ece-b520-c692dfa501cd_isr_871041_56.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c6393991-0ddb-4ece-b520-c692dfa501cd_isr_871041_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:36 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bea3ef6ae242ece06040fa3385ba0be4 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 0b1c26de8d199f833712541c80cbfce8 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5f01e8dc-967f-4a30-aa6b-2f3e3009375c_characterizeImage_871043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5f01e8dc-967f-4a30-aa6b-2f3e3009375c_characterizeImage_871043_30.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2baf73b3-5b33-494c-a22b-9eb6d594a5d2_isr_871043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2baf73b3-5b33-494c-a22b-9eb6d594a5d2_isr_871043_30.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e51250acdf07e281c2636a2b4dd1888 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash dc0d747160a3a9aeb654cdc9853590d3 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3a348a34-0c5e-44d5-9666-132e284de478_characterizeImage_871043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3a348a34-0c5e-44d5-9666-132e284de478_characterizeImage_871043_13.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/93cbfb0b-ac12-4235-85a0-f94b79493654_isr_871043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/93cbfb0b-ac12-4235-85a0-f94b79493654_isr_871043_13.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2267179c3aac868a1302d7671f99e67 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 76fc6526a47fee747318aae8d3005cd3 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6248573f-9bd9-40cf-b556-2314f5d1fd51_characterizeImage_871043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6248573f-9bd9-40cf-b556-2314f5d1fd51_characterizeImage_871043_22.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/220ac616-7ce6-482b-8566-fbf110e53f91_isr_871043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/220ac616-7ce6-482b-8566-fbf110e53f91_isr_871043_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:37 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:37 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fc7405453807d4a0bd0ba1f21f3a752 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 3e251a16fa2dccbcaa0b41ac5e092d4c parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/025cf829-65b5-419a-860d-b849bb8720a0_characterizeImage_871042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/025cf829-65b5-419a-860d-b849bb8720a0_characterizeImage_871042_42.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dcfdaa38-5861-42f1-8d82-ce5fb7c0689b_isr_871042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dcfdaa38-5861-42f1-8d82-ce5fb7c0689b_isr_871042_42.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8220b3f93df738efac10ad8f5829b288 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash a4b2bc5984a418c78be0ea8acadc08a1 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/24fd9d7b-3c5b-44ea-8020-a56a650d0747_characterizeImage_871041_54.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/24fd9d7b-3c5b-44ea-8020-a56a650d0747_characterizeImage_871041_54.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/73d3f025-5968-44cd-a321-81eb7972499d_isr_871041_54.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/73d3f025-5968-44cd-a321-81eb7972499d_isr_871041_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:37 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 169da8398c481bb17c255b82e97dc863 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash cf6b679da1c8ae22f2b38dd0c93a6ed3 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48ce5100-01d1-4636-a962-9e8ec11e8b68_characterizeImage_871043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48ce5100-01d1-4636-a962-9e8ec11e8b68_characterizeImage_871043_4.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/cec61354-9d52-4f11-8286-8ca7bddfdfa2_isr_871043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/cec61354-9d52-4f11-8286-8ca7bddfdfa2_isr_871043_4.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa81fb2f692844ae2fc8349a4d375469 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 7b47fd958fa595fc7e1d358c43d3ceee parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/499b4af1-200d-4519-b2e2-6a38c7e87461_characterizeImage_871043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/499b4af1-200d-4519-b2e2-6a38c7e87461_characterizeImage_871043_58.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9d265c59-223a-4931-9894-e4e0a55f7660_isr_871043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9d265c59-223a-4931-9894-e4e0a55f7660_isr_871043_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:38 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ce48325100388de12767d3ac5d96668 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 91aa3546ef69dbc08539f2c595f075c7 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/25ad6b98-65be-4dc2-94f7-b24d24bb1afc_characterizeImage_871042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/25ad6b98-65be-4dc2-94f7-b24d24bb1afc_characterizeImage_871042_33.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a8bf1a77-e592-411c-b752-01ac151cafb6_isr_871042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a8bf1a77-e592-411c-b752-01ac151cafb6_isr_871042_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:39 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6377abcd603a86a125ce67c616b5d435 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash dc45bf249664153c3d51798bf86350a1 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1eb01151-f8ce-4a96-9acd-adea7cf2799e_characterizeImage_871043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1eb01151-f8ce-4a96-9acd-adea7cf2799e_characterizeImage_871043_49.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/4e727cfe-2f09-4ed3-9d6a-01e099ae6212_isr_871043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/4e727cfe-2f09-4ed3-9d6a-01e099ae6212_isr_871043_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:39 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:39 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:40 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:41 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:41 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:42 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1848d19afeaf4d81f6c686aba1ad259 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash b9f25b8050424a32e061719ffddc3398 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6ad25ea3-e458-4892-8e63-d28893e4856e_characterizeImage_871041_55.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6ad25ea3-e458-4892-8e63-d28893e4856e_characterizeImage_871041_55.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ffa6a6ab-e8aa-4ee2-9170-76a7e10bb9ab_isr_871041_55.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ffa6a6ab-e8aa-4ee2-9170-76a7e10bb9ab_isr_871041_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:42 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:43 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:43 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:44 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:44 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:45 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:45 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bafcc695d7ab0207ef0882672e5cf489 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 47afe437c29c8c0c9fa31a683982f876 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eafd8dac-98fb-4695-9762-5e011e1a2db7_characterizeImage_871042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eafd8dac-98fb-4695-9762-5e011e1a2db7_characterizeImage_871042_45.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a55f7de3-91af-4082-b3b9-36803bfa6d91_isr_871042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a55f7de3-91af-4082-b3b9-36803bfa6d91_isr_871042_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:46 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:46 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:47 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f80f2f14d829ffee596424a8ab0df3f with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 15c2f068e1d7b6e1e384918b134cfe3f parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7619ae17-1afd-43f4-bc8b-f576de30cff3_characterizeImage_871043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7619ae17-1afd-43f4-bc8b-f576de30cff3_characterizeImage_871043_5.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/7c8f619d-913c-440d-8a34-1408a428a950_isr_871043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/7c8f619d-913c-440d-8a34-1408a428a950_isr_871043_5.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56e00ca27074402d068cfc99a2eabd9a with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 89023e2feb5156f67ca0a905dba6aa9a parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/cd35785d-3d68-40ab-81fb-847203680fff_characterizeImage_871041_31.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/cd35785d-3d68-40ab-81fb-847203680fff_characterizeImage_871041_31.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b9275f9d-3132-40e2-876f-5d3d34a0f321_isr_871041_31.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b9275f9d-3132-40e2-876f-5d3d34a0f321_isr_871041_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:48 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:48 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1e986b752be37ce90c740bf41fa023e with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 5cacc6d1b110bf9a50dfba33968ffba9 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/988a0658-7d46-4208-aa30-868ad6b811e3_characterizeImage_871042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/988a0658-7d46-4208-aa30-868ad6b811e3_characterizeImage_871042_38.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e70d7f1-995c-472c-8e31-806ac6a5187c_isr_871042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e70d7f1-995c-472c-8e31-806ac6a5187c_isr_871042_38.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415d40fb1679696d4be8424d3130aef8 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash b8d73e0febb2ec75c5711b82724fad07 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9346d03e-982d-4bb0-973c-2c7066b9abc4_characterizeImage_871043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9346d03e-982d-4bb0-973c-2c7066b9abc4_characterizeImage_871043_45.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9282c598-d4c8-4d55-b21a-929dd3848fb0_isr_871043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9282c598-d4c8-4d55-b21a-929dd3848fb0_isr_871043_45.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de65522d279232f3ea77c197254cc26 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 6424445b259375aa0df4fa08ebd8e5af parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/96d62149-660b-422d-aff2-1c7293bacfa5_characterizeImage_871042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/96d62149-660b-422d-aff2-1c7293bacfa5_characterizeImage_871042_50.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7bcd3478-f1d2-4f93-8e64-273416673dad_isr_871042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7bcd3478-f1d2-4f93-8e64-273416673dad_isr_871042_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:49 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97c0868f11c1409ff11b3740527113c5 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 50731f48795193a5337973658d2aa571 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/c1aed269-d6ac-470a-b039-0d89bca7e01b_characterizeImage_871042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/c1aed269-d6ac-470a-b039-0d89bca7e01b_characterizeImage_871042_21.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9aaa0724-b59c-44b7-82ed-b879ef201424_isr_871042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/9aaa0724-b59c-44b7-82ed-b879ef201424_isr_871042_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:49 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:50 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:50 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f08a6a6317e6a0880cae9148cff6c7a4 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash abe93261017836899b8e58756410ca97 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1952b486-0c6a-41bb-b709-8f06c1e35a08_characterizeImage_871043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1952b486-0c6a-41bb-b709-8f06c1e35a08_characterizeImage_871043_27.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78a8ad1d-3da5-4b4e-9af8-29fbe5fb1f71_isr_871043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78a8ad1d-3da5-4b4e-9af8-29fbe5fb1f71_isr_871043_27.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd77c9580acf5dc9f6cea8804951aefd with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash d13294bb73a7f95b4613bea86498ede2 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/453b802f-3250-4af9-a4ac-069eac939c4c_characterizeImage_871042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/453b802f-3250-4af9-a4ac-069eac939c4c_characterizeImage_871042_4.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a9cfec61-8cdc-408c-b338-82bcc88e9573_isr_871042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a9cfec61-8cdc-408c-b338-82bcc88e9573_isr_871042_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:51 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf2f243bff2efb14cf899e9352c20a08 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash f5eae74585177c609e8bf042bb3ebbe9 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9242c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5b0e7447-e2ae-4c17-a970-511c2ab1e802_characterizeImage_871042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5b0e7447-e2ae-4c17-a970-511c2ab1e802_characterizeImage_871042_40.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/89616ef1-3eb5-48bd-b2d1-2fb4f894140a_isr_871042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/89616ef1-3eb5-48bd-b2d1-2fb4f894140a_isr_871042_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:52 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58cd0675edf820db9fd59fb14c17c2b with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash c33884a77316dfbf5e54e2d99dbdbb03 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a51e38a5-eecf-4c54-8967-bde0ea9cf73d_characterizeImage_871043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a51e38a5-eecf-4c54-8967-bde0ea9cf73d_characterizeImage_871043_57.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e33a43c-21e8-4446-9759-2f33c634773e_isr_871043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e33a43c-21e8-4446-9759-2f33c634773e_isr_871043_57.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b79f53faa40c04a0c702634e15598a8 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 2823898019cc23a54caddcfc5b66d27f parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9245e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/804d236c-cb2c-4153-b647-66d857c9b82b_characterizeImage_871042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/804d236c-cb2c-4153-b647-66d857c9b82b_characterizeImage_871042_54.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/fa4a2996-4ba2-4c52-aff9-704dcac2c758_isr_871042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/fa4a2996-4ba2-4c52-aff9-704dcac2c758_isr_871042_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:52 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b5ff52b2e8c8b6459a926d1b5323f8 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash aa6be01026a5b44dc088c08084248206 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d2a9411d-a166-4271-8dc7-731763a0d1f4_characterizeImage_871042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d2a9411d-a166-4271-8dc7-731763a0d1f4_characterizeImage_871042_20.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c01a6f11-cb57-4b19-937d-8e40904e709e_isr_871042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c01a6f11-cb57-4b19-937d-8e40904e709e_isr_871042_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:53 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:53 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:54 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:54 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:54 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad8d1217ece84d9303a5f6ba22530827 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 032d37eb588aacf0f1f132e821469dee parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a3b9f216-87a3-4b1f-b077-d6a5624351ed_characterizeImage_871041_5.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a3b9f216-87a3-4b1f-b077-d6a5624351ed_characterizeImage_871041_5.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3e88d2a7-4715-47f5-9afa-c747ebeb627c_isr_871041_5.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3e88d2a7-4715-47f5-9afa-c747ebeb627c_isr_871041_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:55 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 36/11 running/pending blocks, and 37 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:24:55 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:24:55 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e6b441a111174d99b6d60300df1d2d with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 0daaf83e095089d9c7b4eb2a63cf9e5a parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9274c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fcecbba5-81e1-4ebe-a890-69e47865e6dc_characterizeImage_871041_32.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fcecbba5-81e1-4ebe-a890-69e47865e6dc_characterizeImage_871041_32.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/abeb62e1-4adf-4653-9b18-254a98181800_isr_871041_32.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/abeb62e1-4adf-4653-9b18-254a98181800_isr_871041_32.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8d05472edc56566252ea60d672faf6b with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash f202c284972c2f690614f4d323cbc289 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06e06b0e-dd5e-42bd-8c51-7243aa7315a5_characterizeImage_871043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06e06b0e-dd5e-42bd-8c51-7243aa7315a5_characterizeImage_871043_51.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6802908a-12b0-4cef-bdf1-baa31e478a37_isr_871043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6802908a-12b0-4cef-bdf1-baa31e478a37_isr_871043_51.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fcb896409d215737ad1a9c0c94c8786 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 5433b4937f559fd7b664ef8ee2dd0bfd parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/708e7743-ede0-436d-8de9-f1d6f90d80a3_characterizeImage_871042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/708e7743-ede0-436d-8de9-f1d6f90d80a3_characterizeImage_871042_62.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a90f9e48-ffbf-4c92-8735-5deb9d84cc8b_isr_871042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/a90f9e48-ffbf-4c92-8735-5deb9d84cc8b_isr_871042_62.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3deec5f03131ca0ecc3d69a6851e334a with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 80d4a462fa01f9f5e7da9a2d41112d24 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbdfc51c-2d59-4deb-9dee-fe3253a29fea_characterizeImage_871041_19.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbdfc51c-2d59-4deb-9dee-fe3253a29fea_characterizeImage_871041_19.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0f54af3c-ad64-447b-b04d-681a02a13d37_isr_871041_19.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0f54af3c-ad64-447b-b04d-681a02a13d37_isr_871041_19.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85e67adb88071c9746c9c78a233e8490 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 9729232317879a6ad89127fb0ad08c68 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9013a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ae0dcb34-3c61-4409-8832-ed57d9dea861_characterizeImage_871042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ae0dcb34-3c61-4409-8832-ed57d9dea861_characterizeImage_871042_16.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/d021aa0d-cbba-4373-a0c6-f421f70a4579_isr_871042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/d021aa0d-cbba-4373-a0c6-f421f70a4579_isr_871042_16.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9ce81dbe224ef0c96048b8b12c315f9 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 55c648b514f54ae5abb61c3133b8d187 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cb9bc7fb-31f1-4bb0-a21b-f3041523c4fd_characterizeImage_871042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cb9bc7fb-31f1-4bb0-a21b-f3041523c4fd_characterizeImage_871042_59.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c9c2cd8d-2f35-4dc7-bc95-1b5c4334a2c5_isr_871042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c9c2cd8d-2f35-4dc7-bc95-1b5c4334a2c5_isr_871042_59.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e77336d4d5ff4c69ba2b3820024b38b0 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 38b866960595e64f1b4697687f10e6c9 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9016c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ae48eadb-e8a8-4b64-ab54-dab14d5c8cdc_characterizeImage_871043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ae48eadb-e8a8-4b64-ab54-dab14d5c8cdc_characterizeImage_871043_15.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dcf6c372-39e7-48c9-9707-c15e538602cc_isr_871043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dcf6c372-39e7-48c9-9707-c15e538602cc_isr_871043_15.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bba1ab064568dea175238effdfc0cb14 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 8edb1e8747539f0aadd3123a2a63fe10 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06f503a0-26c3-42c1-b39f-202bf1ba86c3_characterizeImage_871043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06f503a0-26c3-42c1-b39f-202bf1ba86c3_characterizeImage_871043_50.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a08f3702-9d9c-4fe6-ba93-d117c16831f5_isr_871043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a08f3702-9d9c-4fe6-ba93-d117c16831f5_isr_871043_50.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0936358bb241787955b8f3c48e0502e with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash b7a8ba5b79fb4532715eb2d31bd65035 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6719f4b2-9826-4f85-a649-a1d83576cf5c_characterizeImage_871043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6719f4b2-9826-4f85-a649-a1d83576cf5c_characterizeImage_871043_48.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2396315e-2062-4ffd-8606-00bb3601718b_isr_871043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/2396315e-2062-4ffd-8606-00bb3601718b_isr_871043_48.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a78ba5a5028f5cd424d103e3b63e5c with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 5cb87a939975967c4fb65b76139b92e7 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9019e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/4a3eeaf2-3156-4363-9841-e598f8679ada_characterizeImage_871042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/4a3eeaf2-3156-4363-9841-e598f8679ada_characterizeImage_871042_13.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2220fb4-61e9-4f3d-b190-70e4dbd89726_isr_871042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2220fb4-61e9-4f3d-b190-70e4dbd89726_isr_871042_13.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49d12a0043d6c4e1a0c62e996d4ba8cc with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 8cdc72e7934b05454cac1917d1e0381f parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/19d0b14f-6fc2-4894-b333-a3eaf95c7f0e_characterizeImage_871043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/19d0b14f-6fc2-4894-b333-a3eaf95c7f0e_characterizeImage_871043_37.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/23017e1b-098c-4824-99fb-9d0f46a0e82d_isr_871043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/23017e1b-098c-4824-99fb-9d0f46a0e82d_isr_871043_37.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff0ab2a9b67947c99c8b415b4c589f1a with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 20a9347a58c75effadcde86060a3aae8 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/61bf2b5c-50ad-479b-8533-23c9de96e88d_characterizeImage_871042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/61bf2b5c-50ad-479b-8533-23c9de96e88d_characterizeImage_871042_17.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7be79896-061e-43ff-b09c-f2a7546f9a5b_isr_871042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7be79896-061e-43ff-b09c-f2a7546f9a5b_isr_871042_17.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07460f90d480f1125b3606895b7511f6 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 8194f031673046d53df76d5dc530cec0 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eae20bbd-64ae-49f7-ac46-48975175ce51_characterizeImage_871042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eae20bbd-64ae-49f7-ac46-48975175ce51_characterizeImage_871042_24.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3d44c4e3-3ace-4939-9887-75c5a5a81870_isr_871042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3d44c4e3-3ace-4939-9887-75c5a5a81870_isr_871042_24.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd2dae4d8f6bc654a1e69676e1c1a5eb with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 65db2d99c591fd564a948a2eab6efbb6 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9258a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f029a37c-abe0-449e-b76a-743a51c65f85_characterizeImage_871041_36.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f029a37c-abe0-449e-b76a-743a51c65f85_characterizeImage_871041_36.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/24153914-6ad9-46fa-b30e-c9307b52ea8d_isr_871041_36.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/24153914-6ad9-46fa-b30e-c9307b52ea8d_isr_871041_36.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb78165ba0a5b81cb23261070ea6deb3 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash ce8177bb84fb040db7868849d7dab2fa parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9277e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/8a0e3e8a-2a6c-4915-8468-4c178e91fde6_characterizeImage_871042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/8a0e3e8a-2a6c-4915-8468-4c178e91fde6_characterizeImage_871042_35.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dceba791-8cfa-4fc4-a768-490cf2301477_isr_871042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/dceba791-8cfa-4fc4-a768-490cf2301477_isr_871042_35.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75cad09695b06061f24f3f56a2b784ba with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 7f875c4ba2fb691a7fc09ceb02be7a89 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/17273097-ca08-42ea-8388-d0983e323b12_characterizeImage_871041_44.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/17273097-ca08-42ea-8388-d0983e323b12_characterizeImage_871041_44.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c803f21b-658d-4e8d-b99b-d8cb0c4f5e20_isr_871041_44.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c803f21b-658d-4e8d-b99b-d8cb0c4f5e20_isr_871041_44.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1c7e8b358fc6540911f16ec2316435d with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 89d854e622aba0c27dd0ffc798719fa3 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/32dc9614-9e58-458f-acb2-8cdcbc276cec_characterizeImage_871043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/32dc9614-9e58-458f-acb2-8cdcbc276cec_characterizeImage_871043_43.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3a0a3366-93ae-42e9-bfc4-24cf0e82eb3c_isr_871043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3a0a3366-93ae-42e9-bfc4-24cf0e82eb3c_isr_871043_43.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b950742665220a897e0af584ee36966 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash dfe3b5175a66605c1a5971842c0f802d parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3be50570-3f95-4a07-9cdc-b7430a091602_characterizeImage_871043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3be50570-3f95-4a07-9cdc-b7430a091602_characterizeImage_871043_11.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7d80e86-8bce-416b-8300-12d929301eeb_isr_871043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7d80e86-8bce-416b-8300-12d929301eeb_isr_871043_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:58 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af65a5668e16a5a0a86b8a9ae92b2d5a with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash cfbbd2e5050cbf0eeb952cb2db2ae944 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/273d1143-a245-47ec-8621-cae99d080ac3_characterizeImage_871043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/273d1143-a245-47ec-8621-cae99d080ac3_characterizeImage_871043_40.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d4537a00-4064-4f07-899c-198a2dd0120c_isr_871043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d4537a00-4064-4f07-899c-198a2dd0120c_isr_871043_40.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70c7d1f138997c3a17c680e1a808c7ce with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash cd3aaa93fea01f0914914d853252bc05 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd67dac5-316c-47b7-aa2e-a7e131b18850_characterizeImage_871042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd67dac5-316c-47b7-aa2e-a7e131b18850_characterizeImage_871042_61.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/17ae6056-8ae1-4f40-90f8-ca6c6e2cd85e_isr_871042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/17ae6056-8ae1-4f40-90f8-ca6c6e2cd85e_isr_871042_61.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bab6addef5c5099e2c3e8aac6322082 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 8c0716602a63d2fcef8689ff8738ec21 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9271a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/933c08bf-ecc8-42bb-9730-6268de528d97_characterizeImage_871042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/933c08bf-ecc8-42bb-9730-6268de528d97_characterizeImage_871042_7.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/66aae4ba-7598-4b4f-b877-07e01859602f_isr_871042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/66aae4ba-7598-4b4f-b877-07e01859602f_isr_871042_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:58 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b34f79164b6bbf44b2f0dbf910d1ad9 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 1929b56bc48cdea1db2b0b6b5a3e69d2 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/50f4bae0-53e4-459a-be08-3ff156fcfa0e_characterizeImage_871042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/50f4bae0-53e4-459a-be08-3ff156fcfa0e_characterizeImage_871042_9.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f10101eb-a027-47b6-b530-a7336d717c9c_isr_871042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f10101eb-a027-47b6-b530-a7336d717c9c_isr_871042_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:59 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:24:59 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:00 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81cb7e38a8c4e32351379cb6d1c585e8 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 4edf496dbebabb07b71fb0cd400e5414 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/01941f49-59fe-4257-a190-e5c4969eccef_characterizeImage_871041_43.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/01941f49-59fe-4257-a190-e5c4969eccef_characterizeImage_871041_43.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/23497de9-4bd3-4786-bd1e-1d54927706e1_isr_871041_43.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/23497de9-4bd3-4786-bd1e-1d54927706e1_isr_871041_43.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5081d73cf5f0cf4843dd6a2c7cdd2fb with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 140f717840a789f0ca34907c918a4426 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/322d7fd8-b799-4250-a5c6-bdfb4a57de19_characterizeImage_871042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/322d7fd8-b799-4250-a5c6-bdfb4a57de19_characterizeImage_871042_48.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e810016-4ccd-439a-81ab-f62e1e94de40_isr_871042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e810016-4ccd-439a-81ab-f62e1e94de40_isr_871042_48.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca45ab689e51db9bfc1aae4c69c09134 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash c95867df537dac13c9ffcf5405e17c78 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1ae211c0-2f35-432a-8bd1-54969beb161f_characterizeImage_871042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1ae211c0-2f35-432a-8bd1-54969beb161f_characterizeImage_871042_26.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0b9aefea-0eb9-44ed-b871-89eb28672394_isr_871042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0b9aefea-0eb9-44ed-b871-89eb28672394_isr_871042_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:00 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:01 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:02 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:02 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:03 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:03 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28998c9fe183a3014575855704d980c5 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 5f92143abf6354e046908a32ecfb7a52 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/959d1928-e248-4b8e-a266-08212872b33f_characterizeImage_871041_38.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/959d1928-e248-4b8e-a266-08212872b33f_characterizeImage_871041_38.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/17922236-6769-4697-abe2-8ccdbefb261a_isr_871041_38.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/17922236-6769-4697-abe2-8ccdbefb261a_isr_871041_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:04 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2884b2b22e3a3aea79bd7cd344b49d3d with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 402611e5c2326281654108d37b378ece parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b45fa0b3-be19-4672-846e-120aca4c3c46_characterizeImage_871041_49.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b45fa0b3-be19-4672-846e-120aca4c3c46_characterizeImage_871041_49.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/27960f48-a19f-4813-b07c-ef4fe5c43a6f_isr_871041_49.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/27960f48-a19f-4813-b07c-ef4fe5c43a6f_isr_871041_49.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0619eed0dee30b06a770ce72e11b64c with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 3b661feb4eff66aac344496a059a53af parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/077e8da3-ed71-438e-bef0-7ccb622ef938_characterizeImage_871043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/077e8da3-ed71-438e-bef0-7ccb622ef938_characterizeImage_871043_9.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/44363731-28ea-4675-8621-b8af47a7ff7f_isr_871043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/44363731-28ea-4675-8621-b8af47a7ff7f_isr_871043_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980048 PD 16980049 PD 16980050 PD 16980051 PD 16980052 PD 16980053 PD 16980054 PD 16980055 PD 16980056 PD 16980057 PD 16980058 PD 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:04 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7578dddcbcff087b51bab3ac44eb331c with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 1e145274af70471736c32697dc51c710 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9e7fde70-0dfb-487b-b405-fe1c77d7c4e9_characterizeImage_871041_35.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9e7fde70-0dfb-487b-b405-fe1c77d7c4e9_characterizeImage_871041_35.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b83baa86-ce89-407a-9f0f-254ebdb5871f_isr_871041_35.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/b83baa86-ce89-407a-9f0f-254ebdb5871f_isr_871041_35.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d38ebbb0234acdc21014d9d378da31a8 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash e89e98678150f0d59779216cfe49a672 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3f869960-fd4c-4a49-a241-46f968583cff_characterizeImage_871042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3f869960-fd4c-4a49-a241-46f968583cff_characterizeImage_871042_55.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/30149f5a-0282-4f45-8d97-8fb259ce60a5_isr_871042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/30149f5a-0282-4f45-8d97-8fb259ce60a5_isr_871042_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 PD 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:05 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 PD 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:05 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 PD 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:06 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5e99dc1f5e88729c36f7e44ecc4739 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 79358325a97bc58272e6de12e57bf186 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5d91f46c-d855-4cd5-a89b-4b7ac49c8de7_characterizeImage_871043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5d91f46c-d855-4cd5-a89b-4b7ac49c8de7_characterizeImage_871043_42.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78145b76-605c-4880-8477-b705e69fa6d8_isr_871043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/78145b76-605c-4880-8477-b705e69fa6d8_isr_871043_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 PD 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:06 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 PD 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:07 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b49e17f8118acb01e7973ce9f69b882e with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash bc3dda0989444531416a4c33050778a6 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2d111fb4-4c68-45eb-8259-31f23e840258_characterizeImage_871041_42.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2d111fb4-4c68-45eb-8259-31f23e840258_characterizeImage_871041_42.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c68d034d-5c6c-4be2-84a8-dd4f703ae3e7_isr_871041_42.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c68d034d-5c6c-4be2-84a8-dd4f703ae3e7_isr_871041_42.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb9ba62af583bb36e277c3bf50700261 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 24268c6b5dd2ff35219a42b3bc2603e7 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/fa012941-715a-42ab-bf9c-13de3b2db0cb_characterizeImage_871043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/fa012941-715a-42ab-bf9c-13de3b2db0cb_characterizeImage_871043_34.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/67bf4cc8-58d3-437b-8f5f-66d8afc48730_isr_871043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/67bf4cc8-58d3-437b-8f5f-66d8afc48730_isr_871043_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 PD 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:07 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b94ae1bdfe12f48b90fa150152b3c157 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 448c09bb306953ee1acff8de0b247021 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/19200406-9239-4e4e-8ec5-4abd37e3d511_characterizeImage_871042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/19200406-9239-4e4e-8ec5-4abd37e3d511_characterizeImage_871042_15.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e3a5446-fd7e-4cfa-b9f7-4a06fe3a0024_isr_871042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/0e3a5446-fd7e-4cfa-b9f7-4a06fe3a0024_isr_871042_15.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52d605682ceb0606996ede342daf1a5c with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 4230f662cea53ccc9dc96337eb04b91d parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6b417147-bea3-42c1-8af5-656e67227109_characterizeImage_871042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6b417147-bea3-42c1-8af5-656e67227109_characterizeImage_871042_30.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e37fbae-f1dc-4cf7-b930-2fb8450acb6a_isr_871042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1e37fbae-f1dc-4cf7-b930-2fb8450acb6a_isr_871042_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:08 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:08 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:09 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:09 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15d469adbb3a609dab9efc55c97fc014 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 76938ba7a791a7eabdd13e8914335187 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c68ffa9-8a18-492c-bbab-089577119a18_isr_871041_2.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c68ffa9-8a18-492c-bbab-089577119a18_isr_871041_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:10 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59593a56198f851ea90a5b4fd383e598 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 2b09935e179bee1b55ecf1592b484250 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b5c6d5ed-10ac-4019-8e27-847ff5daa8c1_characterizeImage_871042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b5c6d5ed-10ac-4019-8e27-847ff5daa8c1_characterizeImage_871042_51.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/13f0af7c-2e70-429f-92e7-cb091e8f4ec8_isr_871042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/13f0af7c-2e70-429f-92e7-cb091e8f4ec8_isr_871042_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:10 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb320aac630c646832e7907231b4b2e3 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 1bf2f385cae9439ea83ab804e4460e94 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7e16fef-c352-45d6-8604-4fc2c9016fc3_characterizeImage_871042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7e16fef-c352-45d6-8604-4fc2c9016fc3_characterizeImage_871042_28.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/541d5374-8163-4f19-b18f-a6faa8916ade_isr_871042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/541d5374-8163-4f19-b18f-a6faa8916ade_isr_871042_28.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be52345890fddd529beadfc333b6b21d with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash a6b97f85b184edf5c355c57f7318e03c parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8d883eda-4f0b-49a7-b7cd-7ae0c745fcf3_characterizeImage_871041_34.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8d883eda-4f0b-49a7-b7cd-7ae0c745fcf3_characterizeImage_871041_34.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/240d3311-88fb-4347-8479-76af4eb0e17e_isr_871041_34.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/240d3311-88fb-4347-8479-76af4eb0e17e_isr_871041_34.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b7c95a43244a79ed2128f51ea7d0852 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash f98bf29ea52d8200d4e1aa0963763c9a parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fa62d264-0ba9-4fb9-885e-5ac96cf3375b_characterizeImage_871041_25.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fa62d264-0ba9-4fb9-885e-5ac96cf3375b_characterizeImage_871041_25.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/534da3fc-ea2a-49c0-ba5d-14ef88bb7d51_isr_871041_25.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/534da3fc-ea2a-49c0-ba5d-14ef88bb7d51_isr_871041_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:11 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25835397f75c2a67d2378b59382f3326 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash f86173329275cbe8ea1ed097261d5878 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9007775a-a066-486b-82b3-7f94e9a4ed82_characterizeImage_871042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9007775a-a066-486b-82b3-7f94e9a4ed82_characterizeImage_871042_23.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/28e172bf-cda4-49e2-8b62-ea49ad801473_isr_871042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/28e172bf-cda4-49e2-8b62-ea49ad801473_isr_871042_23.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0366a6a8dc493a3ee29ca919e78d9c8d with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash d9ca63d57611da6dbd48ef7a6977736d parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2201840d-8a4e-4e4e-abe3-0b49e2dd491d_characterizeImage_871043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2201840d-8a4e-4e4e-abe3-0b49e2dd491d_characterizeImage_871043_26.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d3bdf060-0e2c-46fb-bb1d-6c1d62d99355_isr_871043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d3bdf060-0e2c-46fb-bb1d-6c1d62d99355_isr_871043_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:12 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13868688c2ecfe22975fd49ef0552113 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 887b20668f7241621d73eab56396bc2f parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/31a980fa-a8b8-401c-a920-cc743775d2d5_characterizeImage_871042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/31a980fa-a8b8-401c-a920-cc743775d2d5_characterizeImage_871042_58.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2063074-0276-480e-8030-32045e7acab9_isr_871042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f2063074-0276-480e-8030-32045e7acab9_isr_871042_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:12 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 825a37c2102db355b0ae651327f00cad with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash f774e9fc16c7fa103e5a7648602d3870 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b636902f-38bb-47f5-8cba-20665b05a129_characterizeImage_871043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b636902f-38bb-47f5-8cba-20665b05a129_characterizeImage_871043_61.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a8a243fa-52a5-425b-9f3f-eaa224c6f89f_isr_871043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a8a243fa-52a5-425b-9f3f-eaa224c6f89f_isr_871043_61.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b992da060f462af85c5d77730e8ad8 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash e52af7d4199208351b3a7bb74d221fbd parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5360dce-028d-49fd-8a44-ecf7e48ea069_characterizeImage_871041_59.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5360dce-028d-49fd-8a44-ecf7e48ea069_characterizeImage_871041_59.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/153f692e-fbb6-41b5-89aa-d5712718ce9e_isr_871041_59.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/153f692e-fbb6-41b5-89aa-d5712718ce9e_isr_871041_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:12 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49ee5cafecd6c358b4b998695f8dbd11 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash d686fcefaddc53fabec4ca38c9574388 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ce58b9c5-1eea-4919-a3fe-d113e6d85a38_characterizeImage_871043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ce58b9c5-1eea-4919-a3fe-d113e6d85a38_characterizeImage_871043_59.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/35be8e8b-2fda-4fc3-94ce-dcea98f15620_isr_871043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/35be8e8b-2fda-4fc3-94ce-dcea98f15620_isr_871043_59.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef81f17dbf2e95d5a313bd9e4c3c31ca with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 1950b8966c3def60eac51ab492e0fce7 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1f480975-99c7-4077-8d9e-08a03a84da4d_characterizeImage_871042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1f480975-99c7-4077-8d9e-08a03a84da4d_characterizeImage_871042_52.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/22496f55-707c-467b-852e-c981f4a5286f_isr_871042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/22496f55-707c-467b-852e-c981f4a5286f_isr_871042_52.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87337af192350f230f4ea7cbc8733333 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash dc4b681f994fa58940c63bb68d964fde parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5a782ce3-ae21-43c2-a460-6179f0f4127d_characterizeImage_871043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5a782ce3-ae21-43c2-a460-6179f0f4127d_characterizeImage_871043_6.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6a531955-1fb1-423e-b11c-41b764934121_isr_871043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6a531955-1fb1-423e-b11c-41b764934121_isr_871043_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:13 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d1d3b1684815adf29e3c728353ea38e with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash d33307b9fa6d963ce1a0e9bdc481134f parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7704e7c0-faa8-425a-935f-d31fe6e20bf7_characterizeImage_871043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7704e7c0-faa8-425a-935f-d31fe6e20bf7_characterizeImage_871043_41.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/47d6bb58-c1f5-4a02-8e40-18ce8f045be1_isr_871043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/47d6bb58-c1f5-4a02-8e40-18ce8f045be1_isr_871043_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:13 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fed6ffd17c08fa9e108056a5e48d7ed with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 218a6008d45437a92c48f9296a65e57f parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/addffb1b-2c7f-4009-b8f2-6e4ba3a99fb9_characterizeImage_871043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/addffb1b-2c7f-4009-b8f2-6e4ba3a99fb9_characterizeImage_871043_52.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/89f8c8fa-1fbf-479b-a303-9c2e8b993659_isr_871043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/89f8c8fa-1fbf-479b-a303-9c2e8b993659_isr_871043_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:14 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:14 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:15 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2fdb7a31185a597b426f1fa70b71f46 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 6b826ef2ad7e301f43b6aa09f0951c18 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/aa98ec03-8cb0-4b5c-a136-4f5400406cee_characterizeImage_871042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/aa98ec03-8cb0-4b5c-a136-4f5400406cee_characterizeImage_871042_6.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b9c9983f-c8fc-4156-b260-d5aabe2208e5_isr_871042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b9c9983f-c8fc-4156-b260-d5aabe2208e5_isr_871042_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:15 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c35d81f5601bfdbd9d58a50bd48cf50d with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash adf11c3079ea09344a105519658e2d2c parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bd701bdf-a7c1-449c-9b3d-289831f3561c_characterizeImage_871042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bd701bdf-a7c1-449c-9b3d-289831f3561c_characterizeImage_871042_3.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e390578c-b8ae-44ae-9cd4-af79900e2da8_isr_871042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e390578c-b8ae-44ae-9cd4-af79900e2da8_isr_871042_3.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2762b5486879eda41756eddcdcb632b5 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 96816438de1fad3387a42b88280267bc parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/4e8c5957-5ced-4be1-92f4-fc8cf3fa12dc_characterizeImage_871043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/4e8c5957-5ced-4be1-92f4-fc8cf3fa12dc_characterizeImage_871043_39.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4d722da-22ec-492b-bfe6-35a2943a92f4_isr_871043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4d722da-22ec-492b-bfe6-35a2943a92f4_isr_871043_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:16 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4e9755d25327d252e470e7d4923423 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 29b9bc51c143101a984b33d7e381ee97 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7a20b1ce-ffb5-4443-b707-ecc74f9e2cb8_characterizeImage_871042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7a20b1ce-ffb5-4443-b707-ecc74f9e2cb8_characterizeImage_871042_32.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4dcd2ec4-14c0-46d8-98ae-36c01e1a7d18_isr_871042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/4dcd2ec4-14c0-46d8-98ae-36c01e1a7d18_isr_871042_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:16 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1ce79b58fb54a3984e2a0d6915645f5 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 979b3351776cd0e2b0fb8674242b0d1d parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/870a16e7-d91e-4585-8b43-d1c31fb2112e_characterizeImage_871041_6.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/870a16e7-d91e-4585-8b43-d1c31fb2112e_characterizeImage_871041_6.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c13709dd-11c1-4cc4-adad-a30f19b64db3_isr_871041_6.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/c13709dd-11c1-4cc4-adad-a30f19b64db3_isr_871041_6.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 064ec82dce9b4c0b9494d83e411e4db4 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 359d3c97df43cdbf3718f7fb24ea3461 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d7d1c392-eaef-4723-b1f2-7d6689b67a94_characterizeImage_871043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d7d1c392-eaef-4723-b1f2-7d6689b67a94_characterizeImage_871043_47.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/05585126-faa4-4907-a361-da2fa67a63d2_isr_871043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/05585126-faa4-4907-a361-da2fa67a63d2_isr_871043_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:17 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:17 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41cd4547cbb9c315cded79e3f6815692 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 1be5450032712ccb25790c8f027b5aca parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ab83830c-2c18-4f30-a18d-55aa72acf747_characterizeImage_871042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ab83830c-2c18-4f30-a18d-55aa72acf747_characterizeImage_871042_46.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c137ba99-b06d-4779-9a72-63182c3f882c_isr_871042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/c137ba99-b06d-4779-9a72-63182c3f882c_isr_871042_46.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 224b45e6560bfd2d2f5063d34d6da0fc with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 523250834be0454e812ff68269a4a9d2 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/930f9d4c-92ca-4502-8cc3-918699c04820_characterizeImage_871043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/930f9d4c-92ca-4502-8cc3-918699c04820_characterizeImage_871043_29.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/679220e4-3750-4816-b9f0-3fad43bda335_isr_871043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/679220e4-3750-4816-b9f0-3fad43bda335_isr_871043_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:18 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b395d3e4e426264f1f46166c3998eaa with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash b70505e874c247bd3c72ded7f1056808 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e16971a9-f157-41f8-ba21-bc93e16767f1_characterizeImage_871043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e16971a9-f157-41f8-ba21-bc93e16767f1_characterizeImage_871043_10.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/f03129ea-a793-42ea-9836-1ddc95680487_isr_871043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/f03129ea-a793-42ea-9836-1ddc95680487_isr_871043_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:19 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42064ab26305ca0d75dd4092b8dd9566 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 83a230873979b52058d0bfe0fd9d111e parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/07c7a5ed-440b-4049-8398-b99fbe0acb2c_characterizeImage_871042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/07c7a5ed-440b-4049-8398-b99fbe0acb2c_characterizeImage_871042_47.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7a82cbbb-ccd9-4a7b-b91d-08b7a90e982d_isr_871042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/7a82cbbb-ccd9-4a7b-b91d-08b7a90e982d_isr_871042_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:19 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 727c82ede8f2cbadf7d35dd7d9263f4e with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash fc6a2fc2a3f83f648a058994ae04ad87 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7ef0f9b4-958c-43d7-acd9-380ac3b886e8_characterizeImage_871041_4.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7ef0f9b4-958c-43d7-acd9-380ac3b886e8_characterizeImage_871041_4.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/18573897-4338-4df9-b8ed-fed85a7e5c45_isr_871041_4.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/18573897-4338-4df9-b8ed-fed85a7e5c45_isr_871041_4.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c4ef386f3417da2c15ee5aa9af035e7 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 66aff2c51bb6d080d19132a248c949ba parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/77e3432f-1f6a-4398-bb64-bc876497db6f_characterizeImage_871041_16.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/77e3432f-1f6a-4398-bb64-bc876497db6f_characterizeImage_871041_16.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/56b468bf-7778-408e-b851-f3a04dc987b9_isr_871041_16.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/56b468bf-7778-408e-b851-f3a04dc987b9_isr_871041_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:20 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf1b5bcfeefad6fbf02984ec3f734f09 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 7a7b0af3bf8fa3e5dc6b8c54a27a3747 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/40c79ce3-c1b2-4bdc-ab73-f30762b9155d_characterizeImage_871041_23.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/40c79ce3-c1b2-4bdc-ab73-f30762b9155d_characterizeImage_871041_23.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ef89b7d5-6bdb-476c-8c75-d36e442728fa_isr_871041_23.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ef89b7d5-6bdb-476c-8c75-d36e442728fa_isr_871041_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:20 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a57ac4107f087ffbde16fe8dcc696ba2 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash dd1daafce6eafc3ffeb70e45bfca3025 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f40e50f1-c740-44af-9ab2-485fde45edf7_characterizeImage_871043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f40e50f1-c740-44af-9ab2-485fde45edf7_characterizeImage_871043_53.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9b5d2b9-a1bb-48d9-98c2-a7f3ab87f587_isr_871043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9b5d2b9-a1bb-48d9-98c2-a7f3ab87f587_isr_871043_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:21 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2336ba03de87a11531b57f17f7603402 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 96bd4d8db8a11c76c398127758866817 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6d3004d9-c5b2-43c8-a96a-f4bf2706024f_characterizeImage_871041_28.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6d3004d9-c5b2-43c8-a96a-f4bf2706024f_characterizeImage_871041_28.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e58deb7e-bfc0-42ed-8903-ae6dcaf23d00_isr_871041_28.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e58deb7e-bfc0-42ed-8903-ae6dcaf23d00_isr_871041_28.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b93aefad1d29e44d4d2ad76f55e59ca with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash accb7800dc29032582f0e94929ee4f39 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/76de81ce-e9bd-444d-8504-30527f2f56e5_characterizeImage_871041_48.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/76de81ce-e9bd-444d-8504-30527f2f56e5_characterizeImage_871041_48.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58ad7c14-5d83-425b-ba9e-9b9e0af979b4_isr_871041_48.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58ad7c14-5d83-425b-ba9e-9b9e0af979b4_isr_871041_48.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a64b7c0ea14e7b0b8512f2eeaf40a65 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 081474e520a884ad12b213d5d4b59c4e parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3db86887-babb-4754-a3cc-480793265311_characterizeImage_871043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3db86887-babb-4754-a3cc-480793265311_characterizeImage_871043_25.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/bb993842-0051-48a9-8d70-813b5f55472f_isr_871043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/bb993842-0051-48a9-8d70-813b5f55472f_isr_871043_25.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b570e39318edca18370385c14d9ee693 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 3f6a69cdb0b00dca7660ae2fc4ce42bd parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd3cb64f-f87c-49fe-90a8-3559d1bf7823_characterizeImage_871042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd3cb64f-f87c-49fe-90a8-3559d1bf7823_characterizeImage_871042_25.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2191922c-bb64-40ad-b251-0e94c86e1733_isr_871042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2191922c-bb64-40ad-b251-0e94c86e1733_isr_871042_25.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba1a8d5699ee4051b200ffb3e85bb1b with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 91d5c6cd3a64cdc80f23c8659a8d9b88 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8c6a8082-fad4-4dda-9154-0346f50b6f86_characterizeImage_871041_46.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8c6a8082-fad4-4dda-9154-0346f50b6f86_characterizeImage_871041_46.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e3c7694a-97b2-4e54-a54b-acb66d4cc97d_isr_871041_46.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/e3c7694a-97b2-4e54-a54b-acb66d4cc97d_isr_871041_46.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0bc304ffa69e991db8ec29341f2a275 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 226df7d44b3787a2e7bd1e17a7f559ce parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/142d70f5-4761-466b-bc37-d8fc184ccc91_characterizeImage_871042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/142d70f5-4761-466b-bc37-d8fc184ccc91_characterizeImage_871042_60.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/18ffb3cb-4758-482c-8091-256e32d64c07_isr_871042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/18ffb3cb-4758-482c-8091-256e32d64c07_isr_871042_60.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46f52306377377b641d3908874b0b01d with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash dc161ce9fce04f75fa42a64a84c1d6fa parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bd59fd6b-b858-483e-93da-cdef66feb94e_characterizeImage_871041_8.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bd59fd6b-b858-483e-93da-cdef66feb94e_characterizeImage_871041_8.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bc6e56e0-1fd3-44f3-b3ef-4deb567ad593_isr_871041_8.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bc6e56e0-1fd3-44f3-b3ef-4deb567ad593_isr_871041_8.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a735168c76983e3e10ff6e3119ea758f with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash a7020941ef1ff7ad5d973d111df96b5a parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bdd6f726-ca44-401a-8973-1ecb39688ca9_characterizeImage_871041_21.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bdd6f726-ca44-401a-8973-1ecb39688ca9_characterizeImage_871041_21.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a233b837-2dbf-49ef-9162-4d35e5b92d42_isr_871041_21.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a233b837-2dbf-49ef-9162-4d35e5b92d42_isr_871041_21.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03160ee5bf882351641093e9d2c2ac77 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 3efda6c84a09acd689758e47380225fe parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d54d512e-1940-4793-9e27-d369e1abde04_characterizeImage_871043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d54d512e-1940-4793-9e27-d369e1abde04_characterizeImage_871043_38.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b51aa6eb-1f6e-494c-a63b-4ed0ca5bea32_isr_871043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b51aa6eb-1f6e-494c-a63b-4ed0ca5bea32_isr_871043_38.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e28a0412ba91a7ab4424cefda862f74 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash b4691b9eba3e3b9dea837a8a6a6615e0 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ab7d0c66-11eb-4d4d-a48a-b093b5d963c5_characterizeImage_871041_60.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ab7d0c66-11eb-4d4d-a48a-b093b5d963c5_characterizeImage_871041_60.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/032eab5d-cbaf-4ba4-b908-a24dfe149f8e_isr_871041_60.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/032eab5d-cbaf-4ba4-b908-a24dfe149f8e_isr_871041_60.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81a1b6709f01531bb301d5ebc5b2935a with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash c9a230f15a996f2140b14a25e20e1405 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/87c50754-6ea6-4487-adb9-9ca263fdd581_characterizeImage_871041_24.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/87c50754-6ea6-4487-adb9-9ca263fdd581_characterizeImage_871041_24.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/31e94397-ffaa-475a-b93f-7ffb6bd18a00_isr_871041_24.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/31e94397-ffaa-475a-b93f-7ffb6bd18a00_isr_871041_24.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138abbbde8a8ca3024a50a145cd1a418 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 62419166a6c911f390aea87041ce06c6 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d8155253-c601-4992-a952-d63d6b461b18_characterizeImage_871041_22.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d8155253-c601-4992-a952-d63d6b461b18_characterizeImage_871041_22.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/dac3f521-c4de-4243-9d60-3fb9f52cba84_isr_871041_22.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/dac3f521-c4de-4243-9d60-3fb9f52cba84_isr_871041_22.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 445fb03c10dbfd617fbd3ab09feabec9 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 6c99a2bf4085617dbc9f4340bd646607 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/c846e4ad-0c37-480f-93da-d064d7740575_characterizeImage_871041_10.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/c846e4ad-0c37-480f-93da-d064d7740575_characterizeImage_871041_10.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7d3b0471-25bb-4da7-9283-658002ba8f7c_isr_871041_10.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7d3b0471-25bb-4da7-9283-658002ba8f7c_isr_871041_10.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d8a2790bd8dff58b2eb72998baeb1c5 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 87551ecde2a259ace38fd9afd9916ef1 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7baac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/1f77c189-40b9-4d8e-916d-6bd2e27776bc_characterizeImage_871041_12.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/1f77c189-40b9-4d8e-916d-6bd2e27776bc_characterizeImage_871041_12.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a2ca3c9b-de23-426b-b84c-5dc904cfa1f1_isr_871041_12.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/a2ca3c9b-de23-426b-b84c-5dc904cfa1f1_isr_871041_12.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64418e77ef583d67b7f9eeb5b0a95c35 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 80e6ba06e6aa659787723e923644335f parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/817bded6-aa6b-4d13-9dd7-cfcee23b8d76_characterizeImage_871041_37.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/817bded6-aa6b-4d13-9dd7-cfcee23b8d76_characterizeImage_871041_37.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c8485cd-c62f-49cc-b26b-22d9a609cefd_isr_871041_37.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/5c8485cd-c62f-49cc-b26b-22d9a609cefd_isr_871041_37.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20beaf8d2a26e23fc75cd40c774ad7b9 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash ae490019f988580acf3ef00ff76aed8c parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48c22037-5da7-416c-a074-a7d9fbd47cbd_characterizeImage_871043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48c22037-5da7-416c-a074-a7d9fbd47cbd_characterizeImage_871043_62.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/709e92b7-d1ec-490a-9069-1a9827d78ca1_isr_871043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/709e92b7-d1ec-490a-9069-1a9827d78ca1_isr_871043_62.stderr 2024-03-10 15:25:26 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cea5136b891939b96ad74d96ed124b5e with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 73e77699a6cbd8c14bcebb3fdcc93fb2 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5c9e171-aaa9-4231-a2d8-dc9ce8f5c35a_characterizeImage_871041_52.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5c9e171-aaa9-4231-a2d8-dc9ce8f5c35a_characterizeImage_871041_52.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/86932859-34e7-4cc5-a8fc-d6b97d81eeac_isr_871041_52.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/86932859-34e7-4cc5-a8fc-d6b97d81eeac_isr_871041_52.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2105a5010e471f102968454a6c8f8b5 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 339ca029eac14ff1a485b3630b52e886 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/050e19fb-2e9b-4a1d-b39e-7d0e4dbda75b_characterizeImage_871043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/050e19fb-2e9b-4a1d-b39e-7d0e4dbda75b_characterizeImage_871043_54.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4b44517-8526-481c-a6e0-02010b9c5421_isr_871043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/e4b44517-8526-481c-a6e0-02010b9c5421_isr_871043_54.stderr 2024-03-10 15:25:27 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:27 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ddf5b24cd10a18e26b81500a59d5d10 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash a141f8369fd6f23da469470c2af13529 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/0ccd6633-0176-4bf9-b7cc-a12800519856_characterizeImage_871042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/0ccd6633-0176-4bf9-b7cc-a12800519856_characterizeImage_871042_37.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/65dddadb-e29e-49a4-9302-eb5550e74fce_isr_871042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/65dddadb-e29e-49a4-9302-eb5550e74fce_isr_871042_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:28 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:28 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12e9935a7a28e0d8b110b8e8455e4fd3 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 61cf3e3b27b78d3b1ee6347b2d5c2ff8 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ffb4b512-adaa-4b33-9f4c-ea5f675c305f_characterizeImage_871042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ffb4b512-adaa-4b33-9f4c-ea5f675c305f_characterizeImage_871042_19.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07fb4eab-4f65-466b-853a-e09f2db9eaa0_isr_871042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07fb4eab-4f65-466b-853a-e09f2db9eaa0_isr_871042_19.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c49e4391d96f5688270ae72ad468b24 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 8d64f11b215ec15ef869746955852cb5 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7e07be96-038a-43d3-9bf3-78c091ad60b5_characterizeImage_871041_58.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7e07be96-038a-43d3-9bf3-78c091ad60b5_characterizeImage_871041_58.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7c5bd345-aaeb-4cd1-b77b-3b3364616f82_isr_871041_58.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7c5bd345-aaeb-4cd1-b77b-3b3364616f82_isr_871041_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:30 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:31 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:31 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:32 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:33 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d12cff384d64b03bb3eaf7a7a6e0efc7 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash ee1ab2e83fdb28f10153877d2260be23 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7faa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7934e613-f53e-4dbe-b4f1-190ac9c748df_characterizeImage_871043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7934e613-f53e-4dbe-b4f1-190ac9c748df_characterizeImage_871043_12.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7a16f9d-2543-4438-8119-a924cd216c0e_isr_871043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d7a16f9d-2543-4438-8119-a924cd216c0e_isr_871043_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:33 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:33 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:34 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:34 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:35 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ee658cba5bbbd195f0f8619ed7e715a with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 8a901239179885e1110d2808b2f9bd30 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b8e21d0d-3fe5-4614-955d-f1914da096e5_characterizeImage_871041_29.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b8e21d0d-3fe5-4614-955d-f1914da096e5_characterizeImage_871041_29.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2e8bd4d7-e8b5-44d9-91f4-657281bb9ea5_isr_871041_29.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2e8bd4d7-e8b5-44d9-91f4-657281bb9ea5_isr_871041_29.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56d3972f47cc8a224c4d334fbb3cb9ee with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 1ed01baaf684b1a2d205ddd26a9956f9 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbc049de-7269-42a3-89ed-134bc72a494f_characterizeImage_871041_11.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbc049de-7269-42a3-89ed-134bc72a494f_characterizeImage_871041_11.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0bc39768-a801-4576-8fc4-fc581f59f30b_isr_871041_11.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0bc39768-a801-4576-8fc4-fc581f59f30b_isr_871041_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:35 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:36 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 582af961332db966351e98fa68620028 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 22c934deb3213dc74e035d9c78e94506 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6cd89715-3cf1-451a-b266-ccada47e99e4_characterizeImage_871043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6cd89715-3cf1-451a-b266-ccada47e99e4_characterizeImage_871043_31.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3179cf69-86e4-42c5-aa27-2e8d98f7b893_isr_871043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3179cf69-86e4-42c5-aa27-2e8d98f7b893_isr_871043_31.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da24ba1d166a4173bef8aaabc03f8c03 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 64738c953a6384b59834102bb3b23a36 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/115754c0-7dc1-446f-8c0f-3ad26f23664c_characterizeImage_871041_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/115754c0-7dc1-446f-8c0f-3ad26f23664c_characterizeImage_871041_14.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/eabe1f84-7b7a-4e4a-8c33-ac54da871bb9_isr_871041_14.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/eabe1f84-7b7a-4e4a-8c33-ac54da871bb9_isr_871041_14.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb148d1c509d7f5ac3442fd4938e1688 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 87d677c79bfb6d240ac5355fbf35cb2a parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f5798468-4806-435c-9e0a-d103e72716be_characterizeImage_871041_7.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f5798468-4806-435c-9e0a-d103e72716be_characterizeImage_871041_7.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2ef82e96-a5da-48cb-a93e-976c5e2bf7be_isr_871041_7.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/2ef82e96-a5da-48cb-a93e-976c5e2bf7be_isr_871041_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:37 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e5e3a1ed9dfec3db790c63666a6add9 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 8112a2aa8f7d67d0f13c80548af0861b parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7cb1390f-a607-4a5b-b970-550bf96e3987_characterizeImage_871042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7cb1390f-a607-4a5b-b970-550bf96e3987_characterizeImage_871042_18.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07e93b91-7d58-4095-b791-6cb78ac3a193_isr_871042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/07e93b91-7d58-4095-b791-6cb78ac3a193_isr_871042_18.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9118327a6fb2649c9174833e9d91cba4 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash f629d53a88ac00deac5b8956266d7869 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8428e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ac9dcad2-3ad4-44f0-ac09-e6a15eaed759_characterizeImage_871041_1.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ac9dcad2-3ad4-44f0-ac09-e6a15eaed759_characterizeImage_871041_1.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0124d4ed-346c-4610-acfc-989fc5961659_isr_871041_1.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/0124d4ed-346c-4610-acfc-989fc5961659_isr_871041_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:37 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e421c277ef8ef6434e1294031584ae6 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 51a503a26d4549124eb1c685ca624441 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2e6b9180-7aa8-499c-b878-f8fd2d68fd5a_characterizeImage_871043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2e6b9180-7aa8-499c-b878-f8fd2d68fd5a_characterizeImage_871043_14.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ce179a3e-1809-4243-ace3-4b2e78469c0a_isr_871043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ce179a3e-1809-4243-ace3-4b2e78469c0a_isr_871043_14.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8e78e3905744f289c4da223283f6202 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 0f48f298e18fd659cb2617f1a3a9e3a1 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/824b966f-d852-46cb-88ec-e63c90b1e459_characterizeImage_871043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/824b966f-d852-46cb-88ec-e63c90b1e459_characterizeImage_871043_3.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c4ebb6cd-b7bd-4500-838b-50a131df732b_isr_871043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c4ebb6cd-b7bd-4500-838b-50a131df732b_isr_871043_3.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df028053cddaaa75fdff34ed19be6c3d with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 2ae54b88174ed6b07bb616d232ccf2b2 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bdd780a6-9c4d-40e4-8903-56805ccda931_characterizeImage_871042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bdd780a6-9c4d-40e4-8903-56805ccda931_characterizeImage_871042_36.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f135a6af-18a1-4884-8a9d-0cfec8f361ba_isr_871042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/f135a6af-18a1-4884-8a9d-0cfec8f361ba_isr_871042_36.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a71b13fba162721bbffaab8ffa9a336 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 6cd51cc0f5c6baf675062fc53b27d46c parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9c0ab8a2-ae6a-4d27-8a1e-e26e76775734_characterizeImage_871042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9c0ab8a2-ae6a-4d27-8a1e-e26e76775734_characterizeImage_871042_1.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3bb7eda1-7d3b-4e21-95c3-5ee8a7aded50_isr_871042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/3bb7eda1-7d3b-4e21-95c3-5ee8a7aded50_isr_871042_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:38 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:38 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b03da42d0f7bc33405bc41623f4c51a9 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 6697a31d1759f10eed5762a6e4087754 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9b592ef9-1850-4693-bbfc-c4bee8de143b_characterizeImage_871041_40.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9b592ef9-1850-4693-bbfc-c4bee8de143b_characterizeImage_871041_40.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ec10aca4-12f6-4a90-8c8a-04cf1158ca6d_isr_871041_40.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ec10aca4-12f6-4a90-8c8a-04cf1158ca6d_isr_871041_40.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f3786614cb45cf2811e645c7556680e with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 82de727c6f4a8907d3afa28992131205 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a458bda8-e7ac-43ad-98d6-0a3d5db6dfd0_characterizeImage_871043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a458bda8-e7ac-43ad-98d6-0a3d5db6dfd0_characterizeImage_871043_44.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dabcad65-3d80-44bb-91a0-2de160008121_isr_871043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/dabcad65-3d80-44bb-91a0-2de160008121_isr_871043_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:39 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:39 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65df6bed839cf6847614b2287b456cb8 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 0b9efdb3e8e539f7c44c9ecaac97eae6 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ac7b3a7e-c87f-41c5-b206-d238cf43bf41_characterizeImage_871043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ac7b3a7e-c87f-41c5-b206-d238cf43bf41_characterizeImage_871043_60.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a29002f2-d78e-4902-bb9b-82d594dd15fb_isr_871043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/a29002f2-d78e-4902-bb9b-82d594dd15fb_isr_871043_60.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b1d06230839f7b36fd99fd2e687c265 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash b2179d40dcd784683db9981e77f1a1f9 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/dc5b9aea-555a-48cc-966c-97c2672e378e_characterizeImage_871043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/dc5b9aea-555a-48cc-966c-97c2672e378e_characterizeImage_871043_46.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ad1cfbd0-0892-41cf-aa55-bc4cbcfa8fcf_isr_871043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/ad1cfbd0-0892-41cf-aa55-bc4cbcfa8fcf_isr_871043_46.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9316bcb30b56729c0afe3d15a6b8c2d9 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash c43baeadeb695ce08684b854b5464828 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9ffc85c5-13f6-4128-a9c0-ac1ccca7b91c_characterizeImage_871043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9ffc85c5-13f6-4128-a9c0-ac1ccca7b91c_characterizeImage_871043_23.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/963c6033-b710-4a09-b4b1-ca018e8f3e1b_isr_871043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/963c6033-b710-4a09-b4b1-ca018e8f3e1b_isr_871043_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:40 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:40 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84288f2cf27167a3befdba9eff691e90 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 3dd26243380ef90b058244b282a0c4b5 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/10532b4c-0dc5-4752-ac66-9b06607521ea_characterizeImage_871041_50.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/10532b4c-0dc5-4752-ac66-9b06607521ea_characterizeImage_871041_50.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/9fc7fbb0-1d04-4221-9d56-9c8aae52059b_isr_871041_50.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/9fc7fbb0-1d04-4221-9d56-9c8aae52059b_isr_871041_50.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3e7e408fe16a66aa4008d8fb81047a1 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash ae8d87d725a2e2efd45dff7fb77457c7 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/47ff830c-3a58-4604-b972-e2e523de2b31_characterizeImage_871041_47.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/47ff830c-3a58-4604-b972-e2e523de2b31_characterizeImage_871041_47.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6b0d3ff7-0d6e-4de3-949d-a3bd8d028f99_isr_871041_47.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6b0d3ff7-0d6e-4de3-949d-a3bd8d028f99_isr_871041_47.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 337ca1b988a337004249793c68a0107d with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 48e7cf4fde0b2c1cf1a30a9eb7d4b5fb parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7cabfa77-9f9f-4dc7-8d33-f1b1854793b8_characterizeImage_871041_39.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7cabfa77-9f9f-4dc7-8d33-f1b1854793b8_characterizeImage_871041_39.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/82a49231-2042-47ad-b96d-553768e160f5_isr_871041_39.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/82a49231-2042-47ad-b96d-553768e160f5_isr_871041_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:41 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71ff5dd16be988f9c1c3c69d80c02891 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 07017ab3a8e78d4357b53d2cd0f46e75 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8409a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3d9f017c-b96e-4ee2-bcfc-814c0caed233_characterizeImage_871042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3d9f017c-b96e-4ee2-bcfc-814c0caed233_characterizeImage_871042_29.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2a1a3598-6e85-44cd-8fab-0723eb036801_isr_871042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2a1a3598-6e85-44cd-8fab-0723eb036801_isr_871042_29.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43019cba7f9e5142e82203aff9bd610 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 9128b761b880b992f7fad75155c45ba0 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8bbf40b9-0074-43f7-9c7a-2fde0b5b2dbe_characterizeImage_871041_45.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8bbf40b9-0074-43f7-9c7a-2fde0b5b2dbe_characterizeImage_871041_45.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58e404b2-3878-4abd-acec-7970568638cb_isr_871041_45.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/58e404b2-3878-4abd-acec-7970568638cb_isr_871041_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:41 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cfc7364c3323f1db6092f67d37ec4ed with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 1b62c5459f459ee1a13c64922dc5d02a parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/82a8a4ab-b7d9-4b09-9d0e-c920788a3cb7_characterizeImage_871043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/82a8a4ab-b7d9-4b09-9d0e-c920788a3cb7_characterizeImage_871043_19.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/01de7f3c-b47c-4994-a7bf-ec13a5ad0233_isr_871043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/01de7f3c-b47c-4994-a7bf-ec13a5ad0233_isr_871043_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:42 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b510c6255e5b27a1045026254911ef40 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash b0b95fdef4cc438552733a2e99c25017 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a4865518-fd7b-4eda-a282-1f8786ede90f_characterizeImage_871041_61.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a4865518-fd7b-4eda-a282-1f8786ede90f_characterizeImage_871041_61.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7729e41a-3d40-481d-b224-9f9217d1649a_isr_871041_61.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/7729e41a-3d40-481d-b224-9f9217d1649a_isr_871041_61.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ab068a5f51b1671626004616d7690fb with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 7be659a578dca3c2e47f7e9441704bf0 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d4df33c8-0da9-4fa0-bf8b-1d3d60ff2225_characterizeImage_871041_27.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d4df33c8-0da9-4fa0-bf8b-1d3d60ff2225_characterizeImage_871041_27.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/464fef7c-3c86-412a-9e6a-7a2bacc47c41_isr_871041_27.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/464fef7c-3c86-412a-9e6a-7a2bacc47c41_isr_871041_27.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1823bcae93db95e75f3e7248bcd05d with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash c9d6dbadd6b8321db28ad320bdabd269 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/77245805-c265-4953-b997-b8873c5da87b_characterizeImage_871042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/77245805-c265-4953-b997-b8873c5da87b_characterizeImage_871042_10.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b17c2b1c-17e1-4d40-8450-d225d4a75bb5_isr_871042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/b17c2b1c-17e1-4d40-8450-d225d4a75bb5_isr_871042_10.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bc19635ebd5150d31f6a7bccd1fa121 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 91ebc74e33e50dba9f95666d8e0dd883 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/43cec55c-5016-450b-876e-602ba678c64d_characterizeImage_871043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/43cec55c-5016-450b-876e-602ba678c64d_characterizeImage_871043_17.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d24c4c77-def9-4592-b8ab-f08b89db0123_isr_871043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/d24c4c77-def9-4592-b8ab-f08b89db0123_isr_871043_17.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2ff7cbe195e68e359dfb9813cc77341 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 5c97a993d06ab51580ab850ca3d11018 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/55ede27d-4502-4c76-9390-e567a4400db3_characterizeImage_871043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/55ede27d-4502-4c76-9390-e567a4400db3_characterizeImage_871043_18.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e3862ae-e909-422c-ad38-d15e52b22474_isr_871043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/3e3862ae-e909-422c-ad38-d15e52b22474_isr_871043_18.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3e09cd40928c80715c09cd4a77d1c10 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 5389c62fcd04689bec5af47ecb624bc4 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/66cefeb5-6f0f-4e50-8f10-64d97e25f197_characterizeImage_871041_9.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/66cefeb5-6f0f-4e50-8f10-64d97e25f197_characterizeImage_871041_9.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3c67408f-304e-4703-bb42-000b3b08a561_isr_871041_9.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/3c67408f-304e-4703-bb42-000b3b08a561_isr_871041_9.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c85fd9b71a869ef270f308629b66761 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 519d9b4bf9c7fe6340188be8bbc0eb6f parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5c01169b-d2e5-42bf-9dd7-e634f449d462_characterizeImage_871043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5c01169b-d2e5-42bf-9dd7-e634f449d462_characterizeImage_871043_16.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/042d4612-b175-47c6-a736-39e45e4adf8d_isr_871043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/042d4612-b175-47c6-a736-39e45e4adf8d_isr_871043_16.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 753aa46626412d702121c8fe9321c953 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 53cc552bae30fd1062520afb77c63bf3 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7536cb80-fc28-438f-bc94-da67946ccce5_characterizeImage_871042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7536cb80-fc28-438f-bc94-da67946ccce5_characterizeImage_871042_22.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/942d2471-8be4-47ee-ae83-77b2cee36e3d_isr_871042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/942d2471-8be4-47ee-ae83-77b2cee36e3d_isr_871042_22.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d49a27396c35a2e8f5416ee1eb32fe00 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 44a43deaa399a1e10d7a7b0fa55453e4 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ad3afc09-7be3-4142-ac02-77ce747b10be_characterizeImage_871042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ad3afc09-7be3-4142-ac02-77ce747b10be_characterizeImage_871042_44.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/ad180198-00b7-4b80-b6c5-063c8edcf97a_isr_871042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/ad180198-00b7-4b80-b6c5-063c8edcf97a_isr_871042_44.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e39ad29f2f8ed1a4c4c305ceccc07b71 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 6268a13ea9a206dd7a4762d14a3922da parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/686d9bf8-fc9b-4dab-9d76-bbcf06e00e84_characterizeImage_871041_20.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/686d9bf8-fc9b-4dab-9d76-bbcf06e00e84_characterizeImage_871041_20.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ae93778e-37cf-4ad9-bff8-d4d62f088ea9_isr_871041_20.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/ae93778e-37cf-4ad9-bff8-d4d62f088ea9_isr_871041_20.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6f9bda40b57bb993a6a033866915231 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash b98f5b9e50b3165ae33ffed631deb501 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/8391be91-4bdf-4380-8640-80068c62c53b_characterizeImage_871043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/8391be91-4bdf-4380-8640-80068c62c53b_characterizeImage_871043_55.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/682493d4-fb8f-4ffb-bd1f-aa83204535f5_isr_871043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/682493d4-fb8f-4ffb-bd1f-aa83204535f5_isr_871043_55.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54219c02b0302d3b0a57ca65eb8e978a with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash eb91a2cbcdbcfb18842003ae889cb368 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b2071292-3a92-4796-afbb-0094873bbf83_characterizeImage_871041_62.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b2071292-3a92-4796-afbb-0094873bbf83_characterizeImage_871041_62.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf95ca21-df4f-43b7-9523-40b7030f991a_isr_871041_62.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bf95ca21-df4f-43b7-9523-40b7030f991a_isr_871041_62.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4ae0cab1f78dc53d7a8a82491be8055 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 04dc4e3db6e55a8dd0b5b042888de5a5 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:42 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/faa9f722-3437-462d-86e8-971f6e866533_isr_871042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/faa9f722-3437-462d-86e8-971f6e866533_isr_871042_2.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba0c1a0eaa012a410e61bed36ee66d68 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 8472b65837b3fc036afdd4847b51e180 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d1acd1de-e32a-4c31-ace3-7907acffa50c_characterizeImage_871042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d1acd1de-e32a-4c31-ace3-7907acffa50c_characterizeImage_871042_43.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2da3e8a4-023c-4eac-86b0-dd611a06a994_isr_871042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/2da3e8a4-023c-4eac-86b0-dd611a06a994_isr_871042_43.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b346687cb06caa5696a6eeb368362b42 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 99b06d8f224c308ca116f339f8ae30a9 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c92c56f7-791f-4b1e-a97f-e3b4bc327c49_characterizeImage_871043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c92c56f7-791f-4b1e-a97f-e3b4bc327c49_characterizeImage_871043_32.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9887a2c9-73ee-4ed1-b8c6-f40454b1f063_isr_871043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/9887a2c9-73ee-4ed1-b8c6-f40454b1f063_isr_871043_32.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 845b1f84163cd90b4c304a35ff415104 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash fc3c568700260696da76b905f3332227 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7d1140d-9ae2-409c-84fa-0ea10b1e5b63_characterizeImage_871042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7d1140d-9ae2-409c-84fa-0ea10b1e5b63_characterizeImage_871042_14.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/82686b52-1297-4e40-9569-626dab7b7cd7_isr_871042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/82686b52-1297-4e40-9569-626dab7b7cd7_isr_871042_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:43 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40a2355b5eb8e326e59499acda6ed5ef with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash b9023e411580f840bad7099f5657689d parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5a4ba913-11e5-4360-a2c6-f9944719340f_characterizeImage_871042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5a4ba913-11e5-4360-a2c6-f9944719340f_characterizeImage_871042_41.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/35e9b0a8-3ad4-4e21-a11d-75a12f4c3e0f_isr_871042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/35e9b0a8-3ad4-4e21-a11d-75a12f4c3e0f_isr_871042_41.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd63cf6ce7791f0584c8c08f23e876fe with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash e3a434ed5881d9d73ca609a127277d0d parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/e96808ae-2a62-4cf1-ae2d-3d9e1ec55374_characterizeImage_871041_51.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/e96808ae-2a62-4cf1-ae2d-3d9e1ec55374_characterizeImage_871041_51.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/30192b0b-29ce-4c11-9fe8-e3790ba0aed9_isr_871041_51.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/30192b0b-29ce-4c11-9fe8-e3790ba0aed9_isr_871041_51.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35b65a4e0ce30e695bafa68aafa3ba1b with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash e9e8fdbdf6189b4f8f349d1a7541b506 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/0886dbae-636c-4efc-9ec7-70c3be3ac70f_characterizeImage_871043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/0886dbae-636c-4efc-9ec7-70c3be3ac70f_characterizeImage_871043_28.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/808ef37f-1c3f-448d-85f4-622d201eeacd_isr_871043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/808ef37f-1c3f-448d-85f4-622d201eeacd_isr_871043_28.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a743efcec1da381cae0a4dd7f75a14c5 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash ed573b51004e97b744989849e5f595b3 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c90f0b9d-6748-4495-9637-45c0c9d52111_characterizeImage_871043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c90f0b9d-6748-4495-9637-45c0c9d52111_characterizeImage_871043_24.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/66593e27-83d5-473f-9e32-b3ab8b1df53c_isr_871043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/66593e27-83d5-473f-9e32-b3ab8b1df53c_isr_871043_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:43 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3a4747dafb3fa455fece4eb80d0ae72 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 0cd2f70acf390f48ed761e15c4a79f09 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/93783b46-1ad3-4cad-b5b0-91dd149eb7bb_characterizeImage_871042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/93783b46-1ad3-4cad-b5b0-91dd149eb7bb_characterizeImage_871042_49.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1dfb427f-ad94-4abd-945f-786681a77102_isr_871042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/1dfb427f-ad94-4abd-945f-786681a77102_isr_871042_49.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0097b7ad94fd2b931caa8d195ecb655 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 69cb1cecf1b1ff632103aa2c9fe4d511 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8436a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:44 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/75749a2d-01e8-43d5-b096-c8654cc3cf56_characterizeImage_871041_41.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/75749a2d-01e8-43d5-b096-c8654cc3cf56_characterizeImage_871041_41.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bb1a6adb-f72c-4b2d-9c26-216afd34c56f_isr_871041_41.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/bb1a6adb-f72c-4b2d-9c26-216afd34c56f_isr_871041_41.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb069c0bc2128ae20e901661cc18552 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 881c6f1c83698c7b7049ca6f001e4821 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7b7b22d1-5757-48f0-a73e-c7281bdf61db_characterizeImage_871043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7b7b22d1-5757-48f0-a73e-c7281bdf61db_characterizeImage_871043_36.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9ac5a7c-510c-4f5f-aff2-9740b1c09c20_isr_871043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/c9ac5a7c-510c-4f5f-aff2-9740b1c09c20_isr_871043_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:44 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db87698e8a66960d6a41edeb5bea322f with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 2dba972fc3521a512d507336845bba9f parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e95a6277-0bd4-4ee7-a392-e9966882389e_characterizeImage_871043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e95a6277-0bd4-4ee7-a392-e9966882389e_characterizeImage_871043_8.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/797fecb2-292f-4c1a-9e1b-abae4350368e_isr_871043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/797fecb2-292f-4c1a-9e1b-abae4350368e_isr_871043_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:45 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:46 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:46 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:47 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61d5994a4ae60414dd81275203960eb8 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash b58a2367f6995ea975a73fe0535326bb parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3f460ad6-1c2d-4ea3-8c04-e5bc38a29508_characterizeImage_871043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3f460ad6-1c2d-4ea3-8c04-e5bc38a29508_characterizeImage_871043_35.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b95a9700-0bf4-4334-ae93-757d23f255ad_isr_871043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/b95a9700-0bf4-4334-ae93-757d23f255ad_isr_871043_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:47 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:47 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:48 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:48 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:48 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:49 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:49 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:49 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae48e6b4992034c49c1c0cca0cf533f8 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash ea77e2813df827e24ec4ba853c007a64 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b193aae3-4570-4fa0-a8eb-dba40771c7e0_characterizeImage_871043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b193aae3-4570-4fa0-a8eb-dba40771c7e0_characterizeImage_871043_20.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6c7a413f-a1f3-42c1-841d-cdae62a449bb_isr_871043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/6c7a413f-a1f3-42c1-841d-cdae62a449bb_isr_871043_20.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3745dedcb73719de9bca8bcc05e29412 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 5b2a818fc20ffe7585ea839e1e24388c parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/32ae3f27-4f3e-4240-971b-0634665a8d21_characterizeImage_871041_15.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/32ae3f27-4f3e-4240-971b-0634665a8d21_characterizeImage_871041_15.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/883278ec-1a3a-47a1-b8ac-e15bb6ccea71_isr_871041_15.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/883278ec-1a3a-47a1-b8ac-e15bb6ccea71_isr_871041_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:50 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 0 failed parsl.dataflow.dflow INFO: Task 10 marked for retry parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash f2ae214773aad8dd10cd776b3005d9f7 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 1 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:25:50 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c696607f8840dae2652b087f38d0a76 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 96a444163c01f529f2759d445b81abd6 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/676fe280-0676-4a2e-b00b-83410bef04f8_characterizeImage_871043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/676fe280-0676-4a2e-b00b-83410bef04f8_characterizeImage_871043_1.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/15e707c0-0252-43c2-a1d7-217de83ce96c_isr_871043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/15e707c0-0252-43c2-a1d7-217de83ce96c_isr_871043_1.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 819f0ff423e4cec340f9d2c9c1a25070 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash cc1b01a0b5b8009836908b4681d3a9e1 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6e82df33-d544-4eb3-886a-1ce4c96ec65d_characterizeImage_871041_17.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6e82df33-d544-4eb3-886a-1ce4c96ec65d_characterizeImage_871041_17.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6ff582d9-cdb0-4cce-a02f-992cbe918914_isr_871041_17.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871041/6ff582d9-cdb0-4cce-a02f-992cbe918914_isr_871041_17.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c84ef4f86d0e880ff05c06ec900a0cf3 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash c850ef743139b6248ec0e3a8daccf0c0 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1962f118-d664-4ab3-a7f3-12b2d08af253_characterizeImage_871043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1962f118-d664-4ab3-a7f3-12b2d08af253_characterizeImage_871043_33.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/12f53ada-c75d-4040-b66f-76f6eba68585_isr_871043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/12f53ada-c75d-4040-b66f-76f6eba68585_isr_871043_33.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6032fed405fa930a70e52e007143576 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash b41afb15487b660d73bdfea54a557026 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5ff696bd-d262-46db-a89c-546c44b6b987_characterizeImage_871042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5ff696bd-d262-46db-a89c-546c44b6b987_characterizeImage_871042_39.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/cd96324c-f7a1-4837-a672-a52e85ca6e0a_isr_871042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/cd96324c-f7a1-4837-a672-a52e85ca6e0a_isr_871042_39.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a99a43d23edbf2f5cf8753c83db5041f with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 982d198a58dbbaa7a046d6d9ea0b5363 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a64c908e-f4af-4254-931f-e55eca41c7a8_characterizeImage_871043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a64c908e-f4af-4254-931f-e55eca41c7a8_characterizeImage_871043_56.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/8abceed8-80ed-4b4b-b05d-36507515e531_isr_871043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/8abceed8-80ed-4b4b-b05d-36507515e531_isr_871043_56.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8841f85630e2e1576076c2bd537df7d4 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 49f9f3d65a00e3f370267e855564deb8 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/728b0881-8e77-4ccb-8d2d-c57729801a0a_characterizeImage_871043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/728b0881-8e77-4ccb-8d2d-c57729801a0a_characterizeImage_871043_7.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/493021eb-fb97-4625-9c12-6f306cf3569f_isr_871043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871043/493021eb-fb97-4625-9c12-6f306cf3569f_isr_871043_7.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77d95e2b30694dba8e33a887844cda79 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 8d5c6ba40ae94e60e84a9443107d2394 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/448eaa63-5f06-4b4d-8ad4-17b6c8ca1811_characterizeImage_871042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/448eaa63-5f06-4b4d-8ad4-17b6c8ca1811_characterizeImage_871042_34.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e618bac0-7a9e-4017-9415-9f221a31c6e2_isr_871042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/isr/871042/e618bac0-7a9e-4017-9415-9f221a31c6e2_isr_871042_34.stderr 2024-03-10 15:26:01 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:26:01 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:03 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:04 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:04 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:05 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:06 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:06 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:07 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:07 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:08 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:08 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:09 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:10 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:10 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:11 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:11 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:12 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:12 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:13 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:13 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:14 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:15 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:15 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:16 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:16 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:17 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:18 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:18 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:19 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:19 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:20 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:20 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:21 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:21 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:22 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:23 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:23 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:24 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:24 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:25 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:26 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:26 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:27 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:27 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:28 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:28 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:26:34 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:26:34 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:36 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:37 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:37 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:38 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:38 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:39 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:39 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:40 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:41 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:41 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:42 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:42 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:43 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:43 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:44 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:45 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:45 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:46 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:46 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:47 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:48 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:48 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:49 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:49 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:50 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:51 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:51 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:52 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:52 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:53 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:53 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:54 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:55 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:55 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:56 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:56 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:57 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:57 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:58 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:59 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:26:59 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:00 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:00 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:01 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:02 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:27:07 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:27:07 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:09 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:10 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:10 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:11 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:11 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:12 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:12 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:13 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:14 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:14 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:15 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:15 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:16 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:16 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:17 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:18 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89023e2feb5156f67ca0a905dba6aa9a with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 1b3b25aea87e317e3e62ed2848a899c9 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5b8b1b1d-3352-4bed-b05f-90ad5af34f0e_calibrate_871041_31.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5b8b1b1d-3352-4bed-b05f-90ad5af34f0e_calibrate_871041_31.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/cd35785d-3d68-40ab-81fb-847203680fff_characterizeImage_871041_31.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/cd35785d-3d68-40ab-81fb-847203680fff_characterizeImage_871041_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:18 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:19 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:19 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:20 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:21 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:21 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:22 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:22 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:23 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:24 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:24 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:25 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:25 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:26 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:26 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:27 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:28 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:28 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:29 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:29 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:30 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:31 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:31 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:32 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:32 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:33 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:34 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:34 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:35 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:27:40 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:27:41 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:42 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:43 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:43 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:44 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:44 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:45 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:45 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:46 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:46 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:47 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:48 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d32f50f164c0cba68f01765fe2d67cc8 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash ec04b30db85ac020eceb1b6e197aab7e parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f8a4ab6-6beb-4bb5-bd43-93fa5cf392d7_calibrate_871042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f8a4ab6-6beb-4bb5-bd43-93fa5cf392d7_calibrate_871042_31.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/24c456c3-5291-4acf-9a89-c819c3c8ece9_characterizeImage_871042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/24c456c3-5291-4acf-9a89-c819c3c8ece9_characterizeImage_871042_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:48 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:49 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:49 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:50 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:51 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:51 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:52 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:52 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:53 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:53 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:54 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:55 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:55 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:56 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:56 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:57 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:57 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:58 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:58 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:27:59 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:00 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:00 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:01 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:01 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:02 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:02 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:03 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:03 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:04 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:05 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:05 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:06 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2823898019cc23a54caddcfc5b66d27f with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 23015865bf3b250c79cf6f1347a687f1 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/bd5016c7-fc40-444f-9ea1-69044b727b57_calibrate_871042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/bd5016c7-fc40-444f-9ea1-69044b727b57_calibrate_871042_54.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/804d236c-cb2c-4153-b647-66d857c9b82b_characterizeImage_871042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/804d236c-cb2c-4153-b647-66d857c9b82b_characterizeImage_871042_54.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f25b8050424a32e061719ffddc3398 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash b2a57e0cf0643928009baa3980b5f845 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/508b45c0-f8cb-4e5d-a5fa-aa3810206caa_calibrate_871041_55.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/508b45c0-f8cb-4e5d-a5fa-aa3810206caa_calibrate_871041_55.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6ad25ea3-e458-4892-8e63-d28893e4856e_characterizeImage_871041_55.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6ad25ea3-e458-4892-8e63-d28893e4856e_characterizeImage_871041_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:06 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:07 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:28:13 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:28:13 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:15 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:16 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:16 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:17 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:17 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:18 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:19 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:19 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:20 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:20 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:21 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:22 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:22 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:23 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:24 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:24 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:25 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b2bc5984a418c78be0ea8acadc08a1 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash d7d275338d71fa0f99693db27fcf919c parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4016b4c3-036f-4994-aa31-99d241708cc8_calibrate_871041_54.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4016b4c3-036f-4994-aa31-99d241708cc8_calibrate_871041_54.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/24fd9d7b-3c5b-44ea-8020-a56a650d0747_characterizeImage_871041_54.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/24fd9d7b-3c5b-44ea-8020-a56a650d0747_characterizeImage_871041_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:25 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cacc6d1b110bf9a50dfba33968ffba9 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 8f15fd0389ea983c14892581791c5c4b parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ac965867-5a63-459b-849f-fda2baeafe55_calibrate_871042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ac965867-5a63-459b-849f-fda2baeafe55_calibrate_871042_38.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/988a0658-7d46-4208-aa30-868ad6b811e3_characterizeImage_871042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/988a0658-7d46-4208-aa30-868ad6b811e3_characterizeImage_871042_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:26 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:27 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15c2f068e1d7b6e1e384918b134cfe3f with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 6dac40aaec0de89bae5125d6d2e71f03 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/26df321c-1535-48fc-87a0-1b485131237e_calibrate_871043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/26df321c-1535-48fc-87a0-1b485131237e_calibrate_871043_5.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7619ae17-1afd-43f4-bc8b-f576de30cff3_characterizeImage_871043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7619ae17-1afd-43f4-bc8b-f576de30cff3_characterizeImage_871043_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:27 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:28 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:28 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:29 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:30 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:30 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc45bf249664153c3d51798bf86350a1 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 64b5747e887caee9ed0b9cdf51ac2699 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94be30f3-e7f2-46d3-a5f7-21783b4dc34d_calibrate_871043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94be30f3-e7f2-46d3-a5f7-21783b4dc34d_calibrate_871043_49.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1eb01151-f8ce-4a96-9acd-adea7cf2799e_characterizeImage_871043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1eb01151-f8ce-4a96-9acd-adea7cf2799e_characterizeImage_871043_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:31 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50731f48795193a5337973658d2aa571 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 6e56ad504969dc9e7a2a6a664baf9245 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/eaead91d-6044-4c16-83ed-e2c0c18482ae_calibrate_871042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/eaead91d-6044-4c16-83ed-e2c0c18482ae_calibrate_871042_21.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/c1aed269-d6ac-470a-b039-0d89bca7e01b_characterizeImage_871042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/c1aed269-d6ac-470a-b039-0d89bca7e01b_characterizeImage_871042_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:31 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:32 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:33 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:33 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:34 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:34 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:35 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:35 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5eae74585177c609e8bf042bb3ebbe9 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash d6b9cef09205935b145e03dc1c856ee0 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9240e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fc5ae9aa-e42e-4202-9e66-5537da169d58_calibrate_871042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fc5ae9aa-e42e-4202-9e66-5537da169d58_calibrate_871042_40.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5b0e7447-e2ae-4c17-a970-511c2ab1e802_characterizeImage_871042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5b0e7447-e2ae-4c17-a970-511c2ab1e802_characterizeImage_871042_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:36 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:37 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8d73e0febb2ec75c5711b82724fad07 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 34b9a390bb8068277dc8e93a12340029 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/77555292-57b6-47a2-a482-06e534e14c31_calibrate_871043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/77555292-57b6-47a2-a482-06e534e14c31_calibrate_871043_45.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9346d03e-982d-4bb0-973c-2c7066b9abc4_characterizeImage_871043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9346d03e-982d-4bb0-973c-2c7066b9abc4_characterizeImage_871043_45.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15129c944d4e00e319e8c55aaa444397 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 27d70f61e3ddb120acd6c23040395e88 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4352abed-ac8a-47f4-9cee-8fcf4614c5ac_calibrate_871041_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4352abed-ac8a-47f4-9cee-8fcf4614c5ac_calibrate_871041_33.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bcc30fd7-859b-4295-945e-89a2df2171e2_characterizeImage_871041_33.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bcc30fd7-859b-4295-945e-89a2df2171e2_characterizeImage_871041_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:37 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:38 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abe93261017836899b8e58756410ca97 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 1754dab16e5d9a9fcf0cc1315eda3a30 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/643f0dda-1535-4da2-8cae-77584432908d_calibrate_871043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/643f0dda-1535-4da2-8cae-77584432908d_calibrate_871043_27.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1952b486-0c6a-41bb-b709-8f06c1e35a08_characterizeImage_871043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1952b486-0c6a-41bb-b709-8f06c1e35a08_characterizeImage_871043_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:38 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:39 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:40 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b1c26de8d199f833712541c80cbfce8 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash c9a4f0b62850c33ae6ab302edbeb0631 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6c54fc22-0de5-491e-a98e-528f0d58ad20_calibrate_871043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6c54fc22-0de5-491e-a98e-528f0d58ad20_calibrate_871043_30.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5f01e8dc-967f-4a30-aa6b-2f3e3009375c_characterizeImage_871043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5f01e8dc-967f-4a30-aa6b-2f3e3009375c_characterizeImage_871043_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:40 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:41 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:41 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67e3719ce10ceca0b32617113873c516 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 3de3998993fd8fc932a5b1d77085942b parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8be700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/54b44876-67a0-488a-9bc8-4f7341d74a11_calibrate_871042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/54b44876-67a0-488a-9bc8-4f7341d74a11_calibrate_871042_5.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/932df572-b445-4842-8f57-0d943efdb51b_characterizeImage_871042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/932df572-b445-4842-8f57-0d943efdb51b_characterizeImage_871042_5.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 485ce8a3cac4c65599c99090d6af05d9 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 3c0d4d616374cf7b0b0c2402d99efb90 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/52b8465d-a149-4ea3-86a8-ba3e805cd891_calibrate_871041_57.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/52b8465d-a149-4ea3-86a8-ba3e805cd891_calibrate_871041_57.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/73c43a42-039c-46b8-9068-f6d21225a2f3_characterizeImage_871041_57.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/73c43a42-039c-46b8-9068-f6d21225a2f3_characterizeImage_871041_57.stderr 2024-03-10 15:28:47 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:28:47 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:48 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:49 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:49 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:50 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:50 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76fc6526a47fee747318aae8d3005cd3 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 6effe58fb603608714845eedcdfbd49c parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5c62c41f-868c-421c-9ff8-4be6af5a739a_calibrate_871043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5c62c41f-868c-421c-9ff8-4be6af5a739a_calibrate_871043_22.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6248573f-9bd9-40cf-b556-2314f5d1fd51_characterizeImage_871043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6248573f-9bd9-40cf-b556-2314f5d1fd51_characterizeImage_871043_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:51 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:51 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:52 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:53 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:53 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:54 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:54 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:55 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:55 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6424445b259375aa0df4fa08ebd8e5af with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash f58450cbc3ca010d843f0c6fa8fd5ccd parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9253a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2e616136-0832-4cd1-b897-1b16b219222d_calibrate_871042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2e616136-0832-4cd1-b897-1b16b219222d_calibrate_871042_50.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/96d62149-660b-422d-aff2-1c7293bacfa5_characterizeImage_871042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/96d62149-660b-422d-aff2-1c7293bacfa5_characterizeImage_871042_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:56 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:57 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:57 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:58 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:58 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:59 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:28:59 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:00 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:01 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:01 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:02 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:02 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:03 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:03 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:04 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032d37eb588aacf0f1f132e821469dee with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 0e62a04ac9724318b5bfdf800dcf7fcb parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e6022611-33dc-4d9b-9d52-f8e12503637e_calibrate_871041_5.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e6022611-33dc-4d9b-9d52-f8e12503637e_calibrate_871041_5.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a3b9f216-87a3-4b1f-b077-d6a5624351ed_characterizeImage_871041_5.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a3b9f216-87a3-4b1f-b077-d6a5624351ed_characterizeImage_871041_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:05 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:05 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:06 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:06 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:07 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47afe437c29c8c0c9fa31a683982f876 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 67fdc8749e041f55eed45761d7c4c430 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/94763d12-4155-45b5-9e04-5eb5b0513987_calibrate_871042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/94763d12-4155-45b5-9e04-5eb5b0513987_calibrate_871042_45.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eafd8dac-98fb-4695-9762-5e011e1a2db7_characterizeImage_871042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eafd8dac-98fb-4695-9762-5e011e1a2db7_characterizeImage_871042_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:07 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:08 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:09 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:09 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:10 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c33884a77316dfbf5e54e2d99dbdbb03 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 18f0eea5d89b85e8aa008625c9cfb913 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/37c53fa8-45a9-40b3-947b-b8227e54a00c_calibrate_871043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/37c53fa8-45a9-40b3-947b-b8227e54a00c_calibrate_871043_57.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a51e38a5-eecf-4c54-8967-bde0ea9cf73d_characterizeImage_871043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a51e38a5-eecf-4c54-8967-bde0ea9cf73d_characterizeImage_871043_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:10 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f536eb641e71ec62a72698dd92eb8197 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 6d621f1527d1d4b7c7db6d617b556494 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2bba1968-0ad8-442c-9e0b-6cfbae816b14_calibrate_871042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2bba1968-0ad8-442c-9e0b-6cfbae816b14_calibrate_871042_12.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/92caf0bc-fc2e-4592-a894-85c3eec994fe_characterizeImage_871042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/92caf0bc-fc2e-4592-a894-85c3eec994fe_characterizeImage_871042_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:11 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:11 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:12 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:12 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:13 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c032068987cbe0561a27564aeaf28ab with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash b3b503da1a4eb8818ee4cdec096b16f6 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8beb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/220087c0-2650-446d-93a9-69480679784b_calibrate_871041_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/220087c0-2650-446d-93a9-69480679784b_calibrate_871041_53.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2f93ca81-9302-493d-ac1d-e0ee010cf8d8_characterizeImage_871041_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2f93ca81-9302-493d-ac1d-e0ee010cf8d8_characterizeImage_871041_53.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f3f5f5d593d59ae76fe99f20119df27 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 46ea7cd387b8f79ae348a366a1afd3ba parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a7c81b42-3b55-4e3c-b3fe-c18df9459d9e_calibrate_871041_30.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a7c81b42-3b55-4e3c-b3fe-c18df9459d9e_calibrate_871041_30.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5e6605f9-f0d7-4b6a-b0f8-d2c20e12c99a_characterizeImage_871041_30.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5e6605f9-f0d7-4b6a-b0f8-d2c20e12c99a_characterizeImage_871041_30.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf6b679da1c8ae22f2b38dd0c93a6ed3 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash ab831abac9ac73e1c77a96827ffd27d7 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/45947ebc-724d-48c5-a041-5487bf7cc78b_calibrate_871043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/45947ebc-724d-48c5-a041-5487bf7cc78b_calibrate_871043_4.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48ce5100-01d1-4636-a962-9e8ec11e8b68_characterizeImage_871043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48ce5100-01d1-4636-a962-9e8ec11e8b68_characterizeImage_871043_4.stderr 2024-03-10 15:29:18 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:29:19 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5433b4937f559fd7b664ef8ee2dd0bfd with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 844310eb168ede173d904c192b9dd155 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5cffb1e2-a9d7-4e63-8ae6-e122bc612545_calibrate_871042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5cffb1e2-a9d7-4e63-8ae6-e122bc612545_calibrate_871042_62.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/708e7743-ede0-436d-8de9-f1d6f90d80a3_characterizeImage_871042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/708e7743-ede0-436d-8de9-f1d6f90d80a3_characterizeImage_871042_62.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d4bf75d4d0aadaaa9242ce38d458517 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash bd65d2af28f611a95ea33f56d167fd08 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3af544fe-ac85-436a-a9fa-ff720b1f1af6_calibrate_871042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3af544fe-ac85-436a-a9fa-ff720b1f1af6_calibrate_871042_57.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1d9d56f8-a73f-4885-ac46-619095ea8643_characterizeImage_871042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1d9d56f8-a73f-4885-ac46-619095ea8643_characterizeImage_871042_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:21 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:21 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:22 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5063454d9460b32ed243f67b345b36c with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 7078c25f27a5469931af404430e6aa19 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/301518ca-bf5d-47d3-9cad-a7f0fba99a36_calibrate_871041_26.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/301518ca-bf5d-47d3-9cad-a7f0fba99a36_calibrate_871041_26.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5ea73c70-4c2d-4b08-85e5-9c04d4a58826_characterizeImage_871041_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/5ea73c70-4c2d-4b08-85e5-9c04d4a58826_characterizeImage_871041_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:22 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:23 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:23 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:24 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:24 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:25 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:25 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:26 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:26 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:27 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:28 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:28 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:29 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:29 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:30 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:30 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:31 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:32 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:32 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:33 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e251a16fa2dccbcaa0b41ac5e092d4c with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash bf9b4fc87c1bb90542edbbd4f11f3ff5 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1703b98f-033a-4cbe-bd7b-670da71b5e8c_calibrate_871042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1703b98f-033a-4cbe-bd7b-670da71b5e8c_calibrate_871042_42.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/025cf829-65b5-419a-860d-b849bb8720a0_characterizeImage_871042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/025cf829-65b5-419a-860d-b849bb8720a0_characterizeImage_871042_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:33 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:34 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:34 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5c811148fbd580b3ca9d22c12be5dc1 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 870f07df776e08d718671f8f9cff06df parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a1ed9a5b-994c-41cc-9492-5da61687d757_calibrate_871043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a1ed9a5b-994c-41cc-9492-5da61687d757_calibrate_871043_21.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/cc549fc5-960b-4415-8f3f-947a6cb521b9_characterizeImage_871043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/cc549fc5-960b-4415-8f3f-947a6cb521b9_characterizeImage_871043_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:35 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:35 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:36 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:37 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:37 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:38 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:38 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:39 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:39 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:40 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:41 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b47fd958fa595fc7e1d358c43d3ceee with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 2156c66be10418e7bd21f4ba5b5b47ca parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8be660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1317fc37-3cf3-4950-8589-9ff705336e32_calibrate_871043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1317fc37-3cf3-4950-8589-9ff705336e32_calibrate_871043_58.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/499b4af1-200d-4519-b2e2-6a38c7e87461_characterizeImage_871043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/499b4af1-200d-4519-b2e2-6a38c7e87461_characterizeImage_871043_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:41 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:42 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b775a3ea363a3ff67e5f721173967fa with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 08cc2c697daa6679ca3e36a66dc3df3d parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8dd113c0-3dcb-4e23-b7cc-5f406687132e_calibrate_871041_56.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8dd113c0-3dcb-4e23-b7cc-5f406687132e_calibrate_871041_56.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ea87e91e-6ca4-4fec-bd8f-52bc0bb1a427_characterizeImage_871041_56.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ea87e91e-6ca4-4fec-bd8f-52bc0bb1a427_characterizeImage_871041_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:43 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:43 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:44 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:45 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:45 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:46 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d13294bb73a7f95b4613bea86498ede2 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash e3af804e2fd021a424da9665bf78d29c parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb903740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5847f6cd-7c67-40cb-82b3-cc77e1b3fc71_calibrate_871042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5847f6cd-7c67-40cb-82b3-cc77e1b3fc71_calibrate_871042_4.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/453b802f-3250-4af9-a4ac-069eac939c4c_characterizeImage_871042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/453b802f-3250-4af9-a4ac-069eac939c4c_characterizeImage_871042_4.stderr 2024-03-10 15:29:51 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:29:51 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 262 try 0 failed parsl.dataflow.dflow INFO: Task 262 marked for retry parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 76938ba7a791a7eabdd13e8914335187 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 1 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:54 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f202c284972c2f690614f4d323cbc289 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 9edced820766b97bd051686d541529e9 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/56e5b3f0-1bde-4d6c-a1ec-ddb099436fc6_calibrate_871043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/56e5b3f0-1bde-4d6c-a1ec-ddb099436fc6_calibrate_871043_51.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06e06b0e-dd5e-42bd-8c51-7243aa7315a5_characterizeImage_871043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06e06b0e-dd5e-42bd-8c51-7243aa7315a5_characterizeImage_871043_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:55 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:55 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:56 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa6be01026a5b44dc088c08084248206 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 05b1a8ca609f399fe1cacf8dd3c9b206 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9027a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0702634-ffef-4839-a016-ae7810877a7d_calibrate_871042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0702634-ffef-4839-a016-ae7810877a7d_calibrate_871042_20.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d2a9411d-a166-4271-8dc7-731763a0d1f4_characterizeImage_871042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d2a9411d-a166-4271-8dc7-731763a0d1f4_characterizeImage_871042_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:56 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0daaf83e095089d9c7b4eb2a63cf9e5a with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 06c3e16447e0773d7a6e47dd936e4af2 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9272e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/dad9b5af-2d7c-4bd8-bbce-291c0fdc34d3_calibrate_871041_32.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/dad9b5af-2d7c-4bd8-bbce-291c0fdc34d3_calibrate_871041_32.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fcecbba5-81e1-4ebe-a890-69e47865e6dc_characterizeImage_871041_32.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fcecbba5-81e1-4ebe-a890-69e47865e6dc_characterizeImage_871041_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:57 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:58 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:58 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:59 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:29:59 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:00 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:00 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:01 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:02 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee105ba3a8b11ff8c619495f7b34b08e with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 6d3b12ab1a7ed8c09ab0dafa2acb134b parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/beae93a2-62fa-42b4-afdc-bb25ccdb52c7_calibrate_871042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/beae93a2-62fa-42b4-afdc-bb25ccdb52c7_calibrate_871042_8.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9cbedd42-8dcf-4cd1-ad9f-439e55fdf08e_characterizeImage_871042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9cbedd42-8dcf-4cd1-ad9f-439e55fdf08e_characterizeImage_871042_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:02 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:03 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80d4a462fa01f9f5e7da9a2d41112d24 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash a821ee126bd85217056f02749b737079 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b644631c-ef82-41e2-9b1c-1d7151dd4e8b_calibrate_871041_19.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b644631c-ef82-41e2-9b1c-1d7151dd4e8b_calibrate_871041_19.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbdfc51c-2d59-4deb-9dee-fe3253a29fea_characterizeImage_871041_19.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbdfc51c-2d59-4deb-9dee-fe3253a29fea_characterizeImage_871041_19.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9d4564fb3a3ef3a4b04c08c4b87469c with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash c32151581b4a8475549075d33900f94a parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/60c0985f-3cba-43ee-b41e-b90b478d8d60_calibrate_871041_18.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/60c0985f-3cba-43ee-b41e-b90b478d8d60_calibrate_871041_18.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ef03b752-f9bd-4e1c-9953-b740fa34f691_characterizeImage_871041_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ef03b752-f9bd-4e1c-9953-b740fa34f691_characterizeImage_871041_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:03 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:04 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:04 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:05 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:06 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:06 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:07 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:07 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 509ccf68978adc63ee1d8ac594dc954c with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 2e1830df902d8ae96b869db476a87b53 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f5036d46-9b74-4e8d-ba15-e70c1d83bc09_calibrate_871042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f5036d46-9b74-4e8d-ba15-e70c1d83bc09_calibrate_871042_53.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7faddab8-3590-4f94-8ab9-6facc6e139e0_characterizeImage_871042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7faddab8-3590-4f94-8ab9-6facc6e139e0_characterizeImage_871042_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:08 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:08 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aec79722a46162d629833f212753f7cf with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 783ac2d9ef8a970f5b84270820bc49ab parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f640a9c-3cbc-405d-a2db-04f80d087254_calibrate_871042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f640a9c-3cbc-405d-a2db-04f80d087254_calibrate_871042_27.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cdb94df5-74de-4f1c-ae92-194c56f9a7de_characterizeImage_871042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cdb94df5-74de-4f1c-ae92-194c56f9a7de_characterizeImage_871042_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:09 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:09 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:10 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:11 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:11 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:12 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:12 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:13 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:13 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:14 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:15 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:15 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:16 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:16 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:17 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:17 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:18 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:19 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:19 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91aa3546ef69dbc08539f2c595f075c7 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash a3bd63bb4be9fece383ecdf7b1f222f4 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/50c14087-758d-4a2e-981a-e7e484167652_calibrate_871042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/50c14087-758d-4a2e-981a-e7e484167652_calibrate_871042_33.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/25ad6b98-65be-4dc2-94f7-b24d24bb1afc_characterizeImage_871042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/25ad6b98-65be-4dc2-94f7-b24d24bb1afc_characterizeImage_871042_33.stderr 2024-03-10 15:30:25 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:30:25 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9729232317879a6ad89127fb0ad08c68 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash ca972714249dd0054a51420ce71142d4 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9011c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1c107be5-7222-41e6-981b-b93c63a8bf11_calibrate_871042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1c107be5-7222-41e6-981b-b93c63a8bf11_calibrate_871042_16.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ae0dcb34-3c61-4409-8832-ed57d9dea861_characterizeImage_871042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ae0dcb34-3c61-4409-8832-ed57d9dea861_characterizeImage_871042_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:27 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:28 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:28 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:29 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:29 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:30 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:31 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55c648b514f54ae5abb61c3133b8d187 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash e8a8120677bdc4b2d1a1d670773c51db parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb900b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/64af761c-77d1-4f87-b51a-fbd4f8e22da6_calibrate_871042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/64af761c-77d1-4f87-b51a-fbd4f8e22da6_calibrate_871042_59.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cb9bc7fb-31f1-4bb0-a21b-f3041523c4fd_characterizeImage_871042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cb9bc7fb-31f1-4bb0-a21b-f3041523c4fd_characterizeImage_871042_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:31 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:32 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:33 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:33 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:34 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:34 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:35 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7184c3e07fa437ad2f778bc8634fc101 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 2e08ca895ef61e3a243173b074119d84 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/6e27675e-9e99-46ba-ad55-6b0ee9bad1de_calibrate_871042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/6e27675e-9e99-46ba-ad55-6b0ee9bad1de_calibrate_871042_11.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d0e8b56a-9073-4511-ae97-f9626c4957ec_characterizeImage_871042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d0e8b56a-9073-4511-ae97-f9626c4957ec_characterizeImage_871042_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:36 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:36 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:37 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:37 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:38 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:38 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:39 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:40 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:40 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:41 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:41 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:42 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:43 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:43 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:44 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38b866960595e64f1b4697687f10e6c9 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash ea77d9f353869896c25621f58a739904 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9014e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/50a0bb07-5ec9-4c7a-8b06-0be7b97db68f_calibrate_871043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/50a0bb07-5ec9-4c7a-8b06-0be7b97db68f_calibrate_871043_15.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ae48eadb-e8a8-4b64-ab54-dab14d5c8cdc_characterizeImage_871043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ae48eadb-e8a8-4b64-ab54-dab14d5c8cdc_characterizeImage_871043_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:44 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:45 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:45 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:46 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7a8ba5b79fb4532715eb2d31bd65035 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash a623a1fc59dcf3e3bb5508371351fd9a parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb902160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/fa951d4f-5905-4344-ad30-bed71242c60e_calibrate_871043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/fa951d4f-5905-4344-ad30-bed71242c60e_calibrate_871043_48.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6719f4b2-9826-4f85-a649-a1d83576cf5c_characterizeImage_871043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6719f4b2-9826-4f85-a649-a1d83576cf5c_characterizeImage_871043_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:47 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:47 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:48 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:48 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20a9347a58c75effadcde86060a3aae8 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash d662bdb7d93b89c790b550c4e96aec79 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9259e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/87239282-3f61-4332-82d7-437891f930f5_calibrate_871042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/87239282-3f61-4332-82d7-437891f930f5_calibrate_871042_17.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/61bf2b5c-50ad-479b-8533-23c9de96e88d_characterizeImage_871042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/61bf2b5c-50ad-479b-8533-23c9de96e88d_characterizeImage_871042_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:49 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:50 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:50 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:51 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:51 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:52 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:52 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cdc72e7934b05454cac1917d1e0381f with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 522291d61b64542056eb4bcad489e20d parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/88222bc3-7fd6-43f3-8d0e-ec44508c81fe_calibrate_871043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/88222bc3-7fd6-43f3-8d0e-ec44508c81fe_calibrate_871043_37.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/19d0b14f-6fc2-4894-b333-a3eaf95c7f0e_characterizeImage_871043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/19d0b14f-6fc2-4894-b333-a3eaf95c7f0e_characterizeImage_871043_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:30:53 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8194f031673046d53df76d5dc530cec0 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash c5f2e0644892761c2acf846a1b30344c parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb924a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/732f6339-2e90-4e23-a8d2-1191c8e03e2a_calibrate_871042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/732f6339-2e90-4e23-a8d2-1191c8e03e2a_calibrate_871042_24.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eae20bbd-64ae-49f7-ac46-48975175ce51_characterizeImage_871042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/eae20bbd-64ae-49f7-ac46-48975175ce51_characterizeImage_871042_24.stderr 2024-03-10 15:30:58 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:30:59 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:00 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:01 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:01 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:02 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:02 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:03 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:03 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:04 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:05 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:05 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:06 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:07 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:07 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:08 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65db2d99c591fd564a948a2eab6efbb6 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 60830b6ffdceea72ec5aff5480425af3 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb9256c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/08e94fd8-1d08-48a4-a0bd-caa49921aa6f_calibrate_871041_36.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/08e94fd8-1d08-48a4-a0bd-caa49921aa6f_calibrate_871041_36.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f029a37c-abe0-449e-b76a-743a51c65f85_characterizeImage_871041_36.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f029a37c-abe0-449e-b76a-743a51c65f85_characterizeImage_871041_36.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfe3b5175a66605c1a5971842c0f802d with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 9e8db0faebb4fffda1760a0ed23d23e7 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/29085662-19a6-4191-82dd-b965fa0ae2ff_calibrate_871043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/29085662-19a6-4191-82dd-b965fa0ae2ff_calibrate_871043_11.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3be50570-3f95-4a07-9cdc-b7430a091602_characterizeImage_871043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3be50570-3f95-4a07-9cdc-b7430a091602_characterizeImage_871043_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:08 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:09 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:09 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:10 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:11 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:11 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:12 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:12 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:13 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:13 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:14 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:15 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:15 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:16 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:16 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:17 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:17 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:18 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfbbd2e5050cbf0eeb952cb2db2ae944 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 5fee91bc5e91d41a09637fa6934b1e9a parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/28e8b486-e782-4a1d-8faf-b81ee73ff0e2_calibrate_871043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/28e8b486-e782-4a1d-8faf-b81ee73ff0e2_calibrate_871043_40.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/273d1143-a245-47ec-8621-cae99d080ac3_characterizeImage_871043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/273d1143-a245-47ec-8621-cae99d080ac3_characterizeImage_871043_40.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f875c4ba2fb691a7fc09ceb02be7a89 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 526d012be05cb3fb083fcce3fe14e155 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5a0e4199-0b21-49bf-8969-9089c515539d_calibrate_871041_44.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5a0e4199-0b21-49bf-8969-9089c515539d_calibrate_871041_44.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/17273097-ca08-42ea-8388-d0983e323b12_characterizeImage_871041_44.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/17273097-ca08-42ea-8388-d0983e323b12_characterizeImage_871041_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:19 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:19 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:20 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:20 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:21 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:22 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:22 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:23 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89d854e622aba0c27dd0ffc798719fa3 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash e55c3b4f42b70075db5cc511416223ff parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/78feca8d-b1c6-4a25-85df-77c5e4557a2c_calibrate_871043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/78feca8d-b1c6-4a25-85df-77c5e4557a2c_calibrate_871043_43.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/32dc9614-9e58-458f-acb2-8cdcbc276cec_characterizeImage_871043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/32dc9614-9e58-458f-acb2-8cdcbc276cec_characterizeImage_871043_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:23 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:24 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:25 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:25 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4edf496dbebabb07b71fb0cd400e5414 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 64e7a1d361eb292ecaa085fc4c6f213f parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/acc688c3-db11-4c6a-bfb4-7d16bf234521_calibrate_871041_43.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/acc688c3-db11-4c6a-bfb4-7d16bf234521_calibrate_871041_43.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/01941f49-59fe-4257-a190-e5c4969eccef_characterizeImage_871041_43.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/01941f49-59fe-4257-a190-e5c4969eccef_characterizeImage_871041_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:26 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9efe3c94bcf0d70098e6b795eb247444 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 82e31ed8ef647f4efba7fc77dd235c14 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8bd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/105b947b-c483-4333-b46c-076ce908045a_calibrate_871042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/105b947b-c483-4333-b46c-076ce908045a_calibrate_871042_56.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6dd65d2f-063f-45ef-a4dd-176e09d37816_characterizeImage_871042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6dd65d2f-063f-45ef-a4dd-176e09d37816_characterizeImage_871042_56.stderr 2024-03-10 15:31:31 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:31:32 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:33 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:33 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce8177bb84fb040db7868849d7dab2fa with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash eee4a95a0efeb5f98d5d066099f0b973 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c7125700-d86a-488e-964d-6f96f25e76b6_calibrate_871042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c7125700-d86a-488e-964d-6f96f25e76b6_calibrate_871042_35.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/8a0e3e8a-2a6c-4915-8468-4c178e91fde6_characterizeImage_871042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/8a0e3e8a-2a6c-4915-8468-4c178e91fde6_characterizeImage_871042_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:34 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:35 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:35 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:36 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 402611e5c2326281654108d37b378ece with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash dfba67b23f0a49084f58249ab7606883 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c6c68835-4358-428f-a6f0-ecc2f9d766b3_calibrate_871041_49.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c6c68835-4358-428f-a6f0-ecc2f9d766b3_calibrate_871041_49.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b45fa0b3-be19-4672-846e-120aca4c3c46_characterizeImage_871041_49.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b45fa0b3-be19-4672-846e-120aca4c3c46_characterizeImage_871041_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:36 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:37 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:37 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:38 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:39 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c0716602a63d2fcef8689ff8738ec21 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash d4ff3cb0f4e4a186d7fc19dd4efe7202 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a5600f0c-f8e3-4380-b9eb-3729aaf20fe6_calibrate_871042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a5600f0c-f8e3-4380-b9eb-3729aaf20fe6_calibrate_871042_7.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/933c08bf-ecc8-42bb-9730-6268de528d97_characterizeImage_871042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/933c08bf-ecc8-42bb-9730-6268de528d97_characterizeImage_871042_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:39 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:40 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:40 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:41 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:41 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:42 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:43 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:43 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:44 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:44 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:45 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:46 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:46 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:47 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:47 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95867df537dac13c9ffcf5405e17c78 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 48d0559081626a31364af4a9d75c6b04 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c3bbc70e-de57-41fd-8967-3d693d8df533_calibrate_871042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c3bbc70e-de57-41fd-8967-3d693d8df533_calibrate_871042_26.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1ae211c0-2f35-432a-8bd1-54969beb161f_characterizeImage_871042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1ae211c0-2f35-432a-8bd1-54969beb161f_characterizeImage_871042_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:48 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:48 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:49 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:50 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:50 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:51 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:51 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:52 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:53 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:53 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:54 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:54 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:55 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:56 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:56 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:57 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8edb1e8747539f0aadd3123a2a63fe10 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash e5123c3227c4717d8b4b1f145a0e7feb parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/86a7d7a7-08f1-4d09-9cc3-7fb463354e0c_calibrate_871043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/86a7d7a7-08f1-4d09-9cc3-7fb463354e0c_calibrate_871043_50.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06f503a0-26c3-42c1-b39f-202bf1ba86c3_characterizeImage_871043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/06f503a0-26c3-42c1-b39f-202bf1ba86c3_characterizeImage_871043_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:57 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:58 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:31:59 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:32:04 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:32:04 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:05 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:06 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1929b56bc48cdea1db2b0b6b5a3e69d2 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 47f52c10ec6595a3e0fd21aca32da072 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb926660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a2592f92-c30b-459d-9edd-04013cd9b321_calibrate_871042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a2592f92-c30b-459d-9edd-04013cd9b321_calibrate_871042_9.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/50f4bae0-53e4-459a-be08-3ff156fcfa0e_characterizeImage_871042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/50f4bae0-53e4-459a-be08-3ff156fcfa0e_characterizeImage_871042_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:07 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:07 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:08 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:08 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:09 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:09 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:10 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:11 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:11 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:12 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:12 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:13 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:13 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:14 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:15 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:15 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:16 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:16 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:17 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:17 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:18 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:19 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:19 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:20 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:20 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:21 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:21 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:22 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:23 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:23 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:24 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:24 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:25 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:26 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:26 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:27 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:27 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:28 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:28 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:29 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:30 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:30 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:31 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89e98678150f0d59779216cfe49a672 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash a467639dfd43fa106b53c406293fd539 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1845f55-f290-4773-82c2-9d32963a457d_calibrate_871042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1845f55-f290-4773-82c2-9d32963a457d_calibrate_871042_55.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3f869960-fd4c-4a49-a241-46f968583cff_characterizeImage_871042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3f869960-fd4c-4a49-a241-46f968583cff_characterizeImage_871042_55.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 448c09bb306953ee1acff8de0b247021 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 84659e05b80f89133ba4f858cb104411 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1e8ddb26-593d-4035-9651-27ad5dd222ab_calibrate_871042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1e8ddb26-593d-4035-9651-27ad5dd222ab_calibrate_871042_15.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/19200406-9239-4e4e-8ec5-4abd37e3d511_characterizeImage_871042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/19200406-9239-4e4e-8ec5-4abd37e3d511_characterizeImage_871042_15.stderr 2024-03-10 15:32:36 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:32:37 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:38 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:39 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:40 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:40 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:41 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:41 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:42 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:42 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f92143abf6354e046908a32ecfb7a52 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 55ae0ce2fa4df88b2e6109eff63d2d0d parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/f4ef9450-3909-4de5-98bf-06e96c72aec3_calibrate_871041_38.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/f4ef9450-3909-4de5-98bf-06e96c72aec3_calibrate_871041_38.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/959d1928-e248-4b8e-a266-08212872b33f_characterizeImage_871041_38.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/959d1928-e248-4b8e-a266-08212872b33f_characterizeImage_871041_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:43 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:43 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:44 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:45 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:45 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:46 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:46 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:47 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b09935e179bee1b55ecf1592b484250 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 462257214d3ed97854183a43b19e1bd0 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/32a45141-e338-4e31-bc00-02c697e0d156_calibrate_871042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/32a45141-e338-4e31-bc00-02c697e0d156_calibrate_871042_51.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b5c6d5ed-10ac-4019-8e27-847ff5daa8c1_characterizeImage_871042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b5c6d5ed-10ac-4019-8e27-847ff5daa8c1_characterizeImage_871042_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:47 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:48 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:49 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:49 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:50 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:50 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:51 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:52 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:52 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 523250834be0454e812ff68269a4a9d2 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash ec2a06d19f20d6adceac85e80759a373 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b478d0aa-c6f2-416e-940f-cc94979c0880_calibrate_871043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b478d0aa-c6f2-416e-940f-cc94979c0880_calibrate_871043_29.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/930f9d4c-92ca-4502-8cc3-918699c04820_characterizeImage_871043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/930f9d4c-92ca-4502-8cc3-918699c04820_characterizeImage_871043_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:53 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:53 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:54 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:54 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:55 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:56 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:56 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:57 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:57 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:58 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:58 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:32:59 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:00 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc3dda0989444531416a4c33050778a6 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash d8bd81b4978e158b33784a224f9281cc parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7f2d75ab-92e9-4927-835b-eb27ea118bd7_calibrate_871041_42.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7f2d75ab-92e9-4927-835b-eb27ea118bd7_calibrate_871041_42.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2d111fb4-4c68-45eb-8259-31f23e840258_characterizeImage_871041_42.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/2d111fb4-4c68-45eb-8259-31f23e840258_characterizeImage_871041_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:00 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:01 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:01 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:02 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:02 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:03 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:04 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 218a6008d45437a92c48f9296a65e57f with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash e292f6a6ba40049ec8dc91953df1315e parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/9e72670b-77a6-4a03-b5a0-0a1db62570a2_calibrate_871043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/9e72670b-77a6-4a03-b5a0-0a1db62570a2_calibrate_871043_52.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/addffb1b-2c7f-4009-b8f2-6e4ba3a99fb9_characterizeImage_871043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/addffb1b-2c7f-4009-b8f2-6e4ba3a99fb9_characterizeImage_871043_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:33:09 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:33:09 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:11 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29b9bc51c143101a984b33d7e381ee97 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash dccab79c5c9f58ace44f5b42604cf45e parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/04e257e4-402d-43c1-a0f5-5d6c37efd91f_calibrate_871042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/04e257e4-402d-43c1-a0f5-5d6c37efd91f_calibrate_871042_32.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7a20b1ce-ffb5-4443-b707-ecc74f9e2cb8_characterizeImage_871042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7a20b1ce-ffb5-4443-b707-ecc74f9e2cb8_characterizeImage_871042_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:12 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:13 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:13 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24268c6b5dd2ff35219a42b3bc2603e7 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash f26b7d647e635ba10625f9d7d88abc0c parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/90d67a4c-3c5c-43a3-9cb1-f25bb5110cbc_calibrate_871043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/90d67a4c-3c5c-43a3-9cb1-f25bb5110cbc_calibrate_871043_34.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/fa012941-715a-42ab-bf9c-13de3b2db0cb_characterizeImage_871043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/fa012941-715a-42ab-bf9c-13de3b2db0cb_characterizeImage_871043_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:14 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:14 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:15 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:15 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:16 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:17 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:17 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:18 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:18 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:19 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:19 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:20 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:21 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:21 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:22 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:23 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:23 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:24 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:24 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:25 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bf2f385cae9439ea83ab804e4460e94 with result from task 268 parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:26 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 0fd0656e9a4e885bcfc5c6f4a33f1586 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a7d2159c-8acd-4bf0-8bdd-70bd96ddb706_calibrate_871042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a7d2159c-8acd-4bf0-8bdd-70bd96ddb706_calibrate_871042_28.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7e16fef-c352-45d6-8604-4fc2c9016fc3_characterizeImage_871042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7e16fef-c352-45d6-8604-4fc2c9016fc3_characterizeImage_871042_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:26 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:27 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:27 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96816438de1fad3387a42b88280267bc with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 0a1f9c03035eb7b90092d316061f1cee parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3088fb9f-091c-402f-8666-86aa0387b199_calibrate_871043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3088fb9f-091c-402f-8666-86aa0387b199_calibrate_871043_39.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/4e8c5957-5ced-4be1-92f4-fc8cf3fa12dc_characterizeImage_871043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/4e8c5957-5ced-4be1-92f4-fc8cf3fa12dc_characterizeImage_871043_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:28 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc0d747160a3a9aeb654cdc9853590d3 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash f70c239afa1c94a421faf2b934426d9c parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8be2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ccd630f-f644-4b10-8f89-0aa1dafad2e1_calibrate_871043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ccd630f-f644-4b10-8f89-0aa1dafad2e1_calibrate_871043_13.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3a348a34-0c5e-44d5-9666-132e284de478_characterizeImage_871043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3a348a34-0c5e-44d5-9666-132e284de478_characterizeImage_871043_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:29 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:29 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:30 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:30 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:31 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:32 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:32 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:33 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:34 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 140f717840a789f0ca34907c918a4426 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash d512cfe7f6423b21980cc3fc8f217970 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9489d51b-4a40-4420-9354-bb4d40f0813f_calibrate_871042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9489d51b-4a40-4420-9354-bb4d40f0813f_calibrate_871042_48.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/322d7fd8-b799-4250-a5c6-bdfb4a57de19_characterizeImage_871042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/322d7fd8-b799-4250-a5c6-bdfb4a57de19_characterizeImage_871042_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:34 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:35 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:35 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d686fcefaddc53fabec4ca38c9574388 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 63ec31a6fffbc3ed356eaeafef7862b4 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/18550b21-27c5-4ba4-8783-f7135a7a7601_calibrate_871043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/18550b21-27c5-4ba4-8783-f7135a7a7601_calibrate_871043_59.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ce58b9c5-1eea-4919-a3fe-d113e6d85a38_characterizeImage_871043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ce58b9c5-1eea-4919-a3fe-d113e6d85a38_characterizeImage_871043_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:36 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:36 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:37 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:38 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e52af7d4199208351b3a7bb74d221fbd with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 91163fd9e003f1df5c7991ef219032f5 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6e87e311-4768-4651-bba7-197cac8d9353_calibrate_871041_59.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6e87e311-4768-4651-bba7-197cac8d9353_calibrate_871041_59.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5360dce-028d-49fd-8a44-ecf7e48ea069_characterizeImage_871041_59.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5360dce-028d-49fd-8a44-ecf7e48ea069_characterizeImage_871041_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:33:43 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:33:43 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:44 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e145274af70471736c32697dc51c710 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash d810f05031be36ed894298659ca3acd3 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/18509e18-2187-4a71-a582-cc4e96bf6965_calibrate_871041_35.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/18509e18-2187-4a71-a582-cc4e96bf6965_calibrate_871041_35.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9e7fde70-0dfb-487b-b405-fe1c77d7c4e9_characterizeImage_871041_35.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9e7fde70-0dfb-487b-b405-fe1c77d7c4e9_characterizeImage_871041_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:44 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:45 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4230f662cea53ccc9dc96337eb04b91d with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 43d0d8d61ca29ff21214462cf034fd33 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e94d4e64-04ed-4615-b4e9-02f05918abfb_calibrate_871042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e94d4e64-04ed-4615-b4e9-02f05918abfb_calibrate_871042_30.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6b417147-bea3-42c1-8af5-656e67227109_characterizeImage_871042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/6b417147-bea3-42c1-8af5-656e67227109_characterizeImage_871042_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:46 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:46 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:47 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:47 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:48 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:48 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:49 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:50 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:50 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:51 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:51 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:52 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:52 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 359d3c97df43cdbf3718f7fb24ea3461 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 473a3474e8ffce14f5ed4e9f06badcc1 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d9eaa74c-21dc-459d-a816-7665cf57744c_calibrate_871043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d9eaa74c-21dc-459d-a816-7665cf57744c_calibrate_871043_47.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d7d1c392-eaef-4723-b1f2-7d6689b67a94_characterizeImage_871043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d7d1c392-eaef-4723-b1f2-7d6689b67a94_characterizeImage_871043_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:53 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 887b20668f7241621d73eab56396bc2f with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash a21f2f8ab418d1ab13dd13479ce18471 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/dcf2f6a1-f1f2-413d-9e48-07953f5c6e30_calibrate_871042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/dcf2f6a1-f1f2-413d-9e48-07953f5c6e30_calibrate_871042_58.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/31a980fa-a8b8-401c-a920-cc743775d2d5_characterizeImage_871042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/31a980fa-a8b8-401c-a920-cc743775d2d5_characterizeImage_871042_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:53 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:54 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79358325a97bc58272e6de12e57bf186 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 9e7a83341ac61e2c45d9245d8696544d parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb927ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a6e69d86-198e-4ee4-8f7f-c1f8d0e7c3f0_calibrate_871043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a6e69d86-198e-4ee4-8f7f-c1f8d0e7c3f0_calibrate_871043_42.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5d91f46c-d855-4cd5-a89b-4b7ac49c8de7_characterizeImage_871043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5d91f46c-d855-4cd5-a89b-4b7ac49c8de7_characterizeImage_871043_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:54 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:55 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:55 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b661feb4eff66aac344496a059a53af with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash b4b8773f68259f4326cee1a3da3c4737 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/821ee656-ec6e-4034-8ef2-714670fc62ba_calibrate_871043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/821ee656-ec6e-4034-8ef2-714670fc62ba_calibrate_871043_9.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/077e8da3-ed71-438e-bef0-7ccb622ef938_characterizeImage_871043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/077e8da3-ed71-438e-bef0-7ccb622ef938_characterizeImage_871043_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:56 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:57 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:57 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:58 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:58 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:59 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:33:59 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:00 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:01 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:01 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:02 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 979b3351776cd0e2b0fb8674242b0d1d with result from task 289 parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:02 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 94e0000b491b1a3e5464b98535cbe31f parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86852f05-cd6f-45b1-881c-8069576a746a_calibrate_871041_6.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86852f05-cd6f-45b1-881c-8069576a746a_calibrate_871041_6.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/870a16e7-d91e-4585-8b43-d1c31fb2112e_characterizeImage_871041_6.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/870a16e7-d91e-4585-8b43-d1c31fb2112e_characterizeImage_871041_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:03 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:03 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:04 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9ca63d57611da6dbd48ef7a6977736d with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 6d61b6de5633c3fe2dda8ca84b69b25f parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1fb78699-b5c5-4664-a099-48bf2d382bad_calibrate_871043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1fb78699-b5c5-4664-a099-48bf2d382bad_calibrate_871043_26.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2201840d-8a4e-4e4e-abe3-0b49e2dd491d_characterizeImage_871043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2201840d-8a4e-4e4e-abe3-0b49e2dd491d_characterizeImage_871043_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:05 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b70505e874c247bd3c72ded7f1056808 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash abcbd2c3ee7dbe57473de4bcd6408903 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a81fc821-b5d6-4a0a-8009-e0b4d5795382_calibrate_871043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a81fc821-b5d6-4a0a-8009-e0b4d5795382_calibrate_871043_10.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e16971a9-f157-41f8-ba21-bc93e16767f1_characterizeImage_871043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e16971a9-f157-41f8-ba21-bc93e16767f1_characterizeImage_871043_10.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96bd4d8db8a11c76c398127758866817 with result from task 331 parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:05 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 9dc0d81ed168e486206e93b7a8bc5714 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/11638659-f541-4ccc-993b-49cbd3362973_calibrate_871041_28.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/11638659-f541-4ccc-993b-49cbd3362973_calibrate_871041_28.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6d3004d9-c5b2-43c8-a96a-f4bf2706024f_characterizeImage_871041_28.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6d3004d9-c5b2-43c8-a96a-f4bf2706024f_characterizeImage_871041_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:06 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:06 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:07 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:08 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:08 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66aff2c51bb6d080d19132a248c949ba with result from task 307 parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:09 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 62aa34cfb9ffe34eb4926b71440959fa parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ad75767f-0382-451f-8b1e-0f71a4cd2efa_calibrate_871041_16.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ad75767f-0382-451f-8b1e-0f71a4cd2efa_calibrate_871041_16.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/77e3432f-1f6a-4398-bb64-bc876497db6f_characterizeImage_871041_16.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/77e3432f-1f6a-4398-bb64-bc876497db6f_characterizeImage_871041_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adf11c3079ea09344a105519658e2d2c with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 5727bde646810667743199eaee0f7f85 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/0367352f-2141-4d9e-b19e-f5ef1210402e_calibrate_871042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/0367352f-2141-4d9e-b19e-f5ef1210402e_calibrate_871042_3.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bd701bdf-a7c1-449c-9b3d-289831f3561c_characterizeImage_871042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bd701bdf-a7c1-449c-9b3d-289831f3561c_characterizeImage_871042_3.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1950b8966c3def60eac51ab492e0fce7 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash adbb695dc7055d8ed04522c58e10e9b9 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b3980c23-28c2-41be-840e-07b56b942ce3_calibrate_871042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b3980c23-28c2-41be-840e-07b56b942ce3_calibrate_871042_52.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1f480975-99c7-4077-8d9e-08a03a84da4d_characterizeImage_871042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/1f480975-99c7-4077-8d9e-08a03a84da4d_characterizeImage_871042_52.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d33307b9fa6d963ce1a0e9bdc481134f with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 7cbd4763855f405795f6c65dfb23aff3 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f1244a82-ac97-49e7-8a31-e5260af21607_calibrate_871043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f1244a82-ac97-49e7-8a31-e5260af21607_calibrate_871043_41.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7704e7c0-faa8-425a-935f-d31fe6e20bf7_characterizeImage_871043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7704e7c0-faa8-425a-935f-d31fe6e20bf7_characterizeImage_871043_41.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd1daafce6eafc3ffeb70e45bfca3025 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 3741171a85e86dbf523b63cee7912a03 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/e48b4cf6-333a-4323-8b2a-5148ed423d7b_calibrate_871043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/e48b4cf6-333a-4323-8b2a-5148ed423d7b_calibrate_871043_53.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f40e50f1-c740-44af-9ab2-485fde45edf7_characterizeImage_871043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f40e50f1-c740-44af-9ab2-485fde45edf7_characterizeImage_871043_53.stderr 2024-03-10 15:34:14 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:34:14 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc4b681f994fa58940c63bb68d964fde with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash c96ecf2245365a2ce7f40fa979615bc2 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94a2bdf6-3791-49dc-84ee-a9ae190c2a11_calibrate_871043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94a2bdf6-3791-49dc-84ee-a9ae190c2a11_calibrate_871043_6.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5a782ce3-ae21-43c2-a460-6179f0f4127d_characterizeImage_871043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5a782ce3-ae21-43c2-a460-6179f0f4127d_characterizeImage_871043_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:16 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:16 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:17 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:18 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:18 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:19 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be5450032712ccb25790c8f027b5aca with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash a8e00b23733c11452d6bf1773a5239ff parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ed648759-b044-418d-bc86-281db82cb457_calibrate_871042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ed648759-b044-418d-bc86-281db82cb457_calibrate_871042_46.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ab83830c-2c18-4f30-a18d-55aa72acf747_characterizeImage_871042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ab83830c-2c18-4f30-a18d-55aa72acf747_characterizeImage_871042_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:20 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:20 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:21 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:22 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:22 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:23 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:23 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:24 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:25 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:25 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:26 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:26 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6a2fc2a3f83f648a058994ae04ad87 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash a17f4f03b7414b09b24e06818461052b parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/187e0215-ae7d-4805-b7f3-2efade41edc0_calibrate_871041_4.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/187e0215-ae7d-4805-b7f3-2efade41edc0_calibrate_871041_4.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7ef0f9b4-958c-43d7-acd9-380ac3b886e8_characterizeImage_871041_4.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7ef0f9b4-958c-43d7-acd9-380ac3b886e8_characterizeImage_871041_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:27 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:28 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:28 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:29 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:29 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:30 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:30 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83a230873979b52058d0bfe0fd9d111e with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 856a1283fb73c59d2f16740b6c221c1e parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/aada3442-ccf2-4652-b105-72656fa08dc6_calibrate_871042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/aada3442-ccf2-4652-b105-72656fa08dc6_calibrate_871042_47.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/07c7a5ed-440b-4049-8398-b99fbe0acb2c_characterizeImage_871042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/07c7a5ed-440b-4049-8398-b99fbe0acb2c_characterizeImage_871042_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:31 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:32 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:32 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:33 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:34 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:34 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:35 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:35 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:36 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:36 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:37 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:37 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:38 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:39 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:39 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:40 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:40 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:41 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:42 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:42 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b826ef2ad7e301f43b6aa09f0951c18 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash e42e208b802fe08cd686136a0b8a7cfc parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f84be570-9324-4871-be41-b545527d12f0_calibrate_871042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f84be570-9324-4871-be41-b545527d12f0_calibrate_871042_6.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/aa98ec03-8cb0-4b5c-a136-4f5400406cee_characterizeImage_871042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/aa98ec03-8cb0-4b5c-a136-4f5400406cee_characterizeImage_871042_6.stderr 2024-03-10 15:34:48 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:34:48 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:49 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:49 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:50 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:50 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:51 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:51 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:52 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:53 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:53 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:54 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:54 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:55 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:55 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:56 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:57 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:57 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:58 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:58 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:34:59 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:00 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:00 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:01 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:01 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:02 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:03 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:03 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:04 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:04 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:05 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:06 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:06 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:07 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:07 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f98bf29ea52d8200d4e1aa0963763c9a with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 71b1fb38f1ff7fa9cb6d05ff720c0c5e parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b9786428-448c-47f6-8bf0-c06d76bf396b_calibrate_871041_25.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b9786428-448c-47f6-8bf0-c06d76bf396b_calibrate_871041_25.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fa62d264-0ba9-4fb9-885e-5ac96cf3375b_characterizeImage_871041_25.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/fa62d264-0ba9-4fb9-885e-5ac96cf3375b_characterizeImage_871041_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:08 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:08 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:09 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:10 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:10 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:11 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:11 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:12 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:13 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:13 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 081474e520a884ad12b213d5d4b59c4e with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash de92a0d4b8b8cb0b443a11783d5d422d parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/34d6f053-8338-461e-82e7-772a0acc5e1c_calibrate_871043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/34d6f053-8338-461e-82e7-772a0acc5e1c_calibrate_871043_25.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3db86887-babb-4754-a3cc-480793265311_characterizeImage_871043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3db86887-babb-4754-a3cc-480793265311_characterizeImage_871043_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:14 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:14 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc161ce9fce04f75fa42a64a84c1d6fa with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 4035972f91b6c2fbc8ea20d5d869dc5a parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b750007-f860-4c64-9061-f62f6b2817cd_calibrate_871041_8.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b750007-f860-4c64-9061-f62f6b2817cd_calibrate_871041_8.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bd59fd6b-b858-483e-93da-cdef66feb94e_characterizeImage_871041_8.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bd59fd6b-b858-483e-93da-cdef66feb94e_characterizeImage_871041_8.stderr 2024-03-10 15:35:20 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:35:20 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:21 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:21 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:22 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b6fe90ab3a1ab2b4b1ee44d9702cd05 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash a2b11c35f296b351878bb80a7a3b0972 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/71fa787f-2683-41f3-af4a-fdebb3f321ce_calibrate_871041_3.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/71fa787f-2683-41f3-af4a-fdebb3f321ce_calibrate_871041_3.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7afd8e16-fd81-4a8b-afbd-d9f995e7e68a_characterizeImage_871041_3.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7afd8e16-fd81-4a8b-afbd-d9f995e7e68a_characterizeImage_871041_3.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62419166a6c911f390aea87041ce06c6 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash f91e7091431350451f5527dbde3d51c4 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/96ff37f4-69ee-48f6-a8c3-98f56e3f2bf9_calibrate_871041_22.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/96ff37f4-69ee-48f6-a8c3-98f56e3f2bf9_calibrate_871041_22.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d8155253-c601-4992-a952-d63d6b461b18_characterizeImage_871041_22.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d8155253-c601-4992-a952-d63d6b461b18_characterizeImage_871041_22.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f86173329275cbe8ea1ed097261d5878 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 398ffe7b9ccdcbdc7f06a0cfc8e3fe08 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/970fd6b3-2942-49ca-b1cc-0eef4aba4224_calibrate_871042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/970fd6b3-2942-49ca-b1cc-0eef4aba4224_calibrate_871042_23.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9007775a-a066-486b-82b3-7f94e9a4ed82_characterizeImage_871042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9007775a-a066-486b-82b3-7f94e9a4ed82_characterizeImage_871042_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:23 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3efda6c84a09acd689758e47380225fe with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash a101756686a24ed6ea76909637391840 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ff94f95-787e-409c-98a9-c449840e9575_calibrate_871043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ff94f95-787e-409c-98a9-c449840e9575_calibrate_871043_38.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d54d512e-1940-4793-9e27-d369e1abde04_characterizeImage_871043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/d54d512e-1940-4793-9e27-d369e1abde04_characterizeImage_871043_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:23 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:24 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91d5c6cd3a64cdc80f23c8659a8d9b88 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 7ca9ebd0e1f304cfe316c5dd6c085b76 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/64a04a69-cd27-4f32-8947-0d2b5af75896_calibrate_871041_46.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/64a04a69-cd27-4f32-8947-0d2b5af75896_calibrate_871041_46.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8c6a8082-fad4-4dda-9154-0346f50b6f86_characterizeImage_871041_46.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8c6a8082-fad4-4dda-9154-0346f50b6f86_characterizeImage_871041_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:24 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:25 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:26 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:26 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:27 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:27 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:28 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:29 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:29 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:30 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:30 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:31 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:32 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:32 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:33 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:33 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:34 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:35 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:35 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:36 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:36 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:37 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:37 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:38 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:39 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:39 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:40 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:40 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:41 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:41 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:42 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:43 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:43 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:44 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:44 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:45 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:45 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:46 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:46 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7020941ef1ff7ad5d973d111df96b5a with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 0e3a11383abf41de800773ede2f91ca1 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86de14e2-c92a-4e5c-a21b-5a4a56005a85_calibrate_871041_21.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86de14e2-c92a-4e5c-a21b-5a4a56005a85_calibrate_871041_21.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bdd6f726-ca44-401a-8973-1ecb39688ca9_characterizeImage_871041_21.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bdd6f726-ca44-401a-8973-1ecb39688ca9_characterizeImage_871041_21.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6b97f85b184edf5c355c57f7318e03c with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash da207904b94db4b371fd81287515c227 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5f7b10d0-fe92-47ea-a933-03e84b49ecb1_calibrate_871041_34.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5f7b10d0-fe92-47ea-a933-03e84b49ecb1_calibrate_871041_34.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8d883eda-4f0b-49a7-b7cd-7ae0c745fcf3_characterizeImage_871041_34.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8d883eda-4f0b-49a7-b7cd-7ae0c745fcf3_characterizeImage_871041_34.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae490019f988580acf3ef00ff76aed8c with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash dad622b29d6185cad2feb9009639c0ef parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1eac9c0a-ba0f-4b46-b401-af3663136114_calibrate_871043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1eac9c0a-ba0f-4b46-b401-af3663136114_calibrate_871043_62.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48c22037-5da7-416c-a074-a7d9fbd47cbd_characterizeImage_871043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/48c22037-5da7-416c-a074-a7d9fbd47cbd_characterizeImage_871043_62.stderr 2024-03-10 15:35:52 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:35:52 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:53 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73e77699a6cbd8c14bcebb3fdcc93fb2 with result from task 361 parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:53 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 196b03c1d469b2ed69da49b9cdfb691c parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/708fd9e5-aae0-4ba4-933f-687ccd6b350c_calibrate_871041_52.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/708fd9e5-aae0-4ba4-933f-687ccd6b350c_calibrate_871041_52.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5c9e171-aaa9-4231-a2d8-dc9ce8f5c35a_characterizeImage_871041_52.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a5c9e171-aaa9-4231-a2d8-dc9ce8f5c35a_characterizeImage_871041_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:54 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:54 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:55 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:56 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:56 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:57 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:57 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:58 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:59 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:35:59 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry accb7800dc29032582f0e94929ee4f39 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash de52f64b0b6f82838c00a014e35b2031 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ae8b5c80-b37d-45df-9034-8a43dfcced6f_calibrate_871041_48.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ae8b5c80-b37d-45df-9034-8a43dfcced6f_calibrate_871041_48.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/76de81ce-e9bd-444d-8504-30527f2f56e5_characterizeImage_871041_48.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/76de81ce-e9bd-444d-8504-30527f2f56e5_characterizeImage_871041_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:00 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:00 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:01 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:01 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:02 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a901239179885e1110d2808b2f9bd30 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash f9abd400cfbc854c7e3b8b83bcabbf46 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a5a8067d-1d0f-4184-869f-528e4645e396_calibrate_871041_29.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a5a8067d-1d0f-4184-869f-528e4645e396_calibrate_871041_29.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b8e21d0d-3fe5-4614-955d-f1914da096e5_characterizeImage_871041_29.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b8e21d0d-3fe5-4614-955d-f1914da096e5_characterizeImage_871041_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:03 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:03 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:04 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:04 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:05 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:06 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:06 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:07 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:07 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c934deb3213dc74e035d9c78e94506 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 9179727d6a488969b2c69f75ee37e77e parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/15a20ca3-2972-4807-aaad-6da2eb210d3c_calibrate_871043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/15a20ca3-2972-4807-aaad-6da2eb210d3c_calibrate_871043_31.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6cd89715-3cf1-451a-b266-ccada47e99e4_characterizeImage_871043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/6cd89715-3cf1-451a-b266-ccada47e99e4_characterizeImage_871043_31.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4691b9eba3e3b9dea837a8a6a6615e0 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 7cb6b447cd5fb1ac23fa412d3ab78bdf parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d377c233-963f-4f8f-a200-2b0e3e10a583_calibrate_871041_60.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d377c233-963f-4f8f-a200-2b0e3e10a583_calibrate_871041_60.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ab7d0c66-11eb-4d4d-a48a-b093b5d963c5_characterizeImage_871041_60.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ab7d0c66-11eb-4d4d-a48a-b093b5d963c5_characterizeImage_871041_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:08 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:08 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:09 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:10 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:10 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:11 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:11 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:12 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:12 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:13 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:14 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7b0af3bf8fa3e5dc6b8c54a27a3747 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 0a326a75ba4a9c892b720e66be25d997 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6c004ac6-d923-4e18-b8cb-272cd2a9399e_calibrate_871041_23.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6c004ac6-d923-4e18-b8cb-272cd2a9399e_calibrate_871041_23.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/40c79ce3-c1b2-4bdc-ab73-f30762b9155d_characterizeImage_871041_23.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/40c79ce3-c1b2-4bdc-ab73-f30762b9155d_characterizeImage_871041_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:14 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:15 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:15 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:16 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:16 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:17 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:18 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:18 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c99a2bf4085617dbc9f4340bd646607 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash b8c72c1de7c88e893a7ee68418ff42ab parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d2012912-400f-4663-a3dd-219afbbc924f_calibrate_871041_10.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d2012912-400f-4663-a3dd-219afbbc924f_calibrate_871041_10.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/c846e4ad-0c37-480f-93da-d064d7740575_characterizeImage_871041_10.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/c846e4ad-0c37-480f-93da-d064d7740575_characterizeImage_871041_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 481 try 0 failed parsl.dataflow.dflow INFO: Task 481 marked for retry parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 04dc4e3db6e55a8dd0b5b042888de5a5 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 1 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stderr 2024-03-10 15:36:24 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:36:24 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:25 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9a230f15a996f2140b14a25e20e1405 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 2780b9469103f5ce8d948230b1d04794 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/3fed0b54-b165-4715-baf3-8f3ef59a7323_calibrate_871041_24.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/3fed0b54-b165-4715-baf3-8f3ef59a7323_calibrate_871041_24.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/87c50754-6ea6-4487-adb9-9ca263fdd581_characterizeImage_871041_24.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/87c50754-6ea6-4487-adb9-9ca263fdd581_characterizeImage_871041_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:26 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:27 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:27 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:28 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:29 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:29 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:30 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:30 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:31 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:32 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:32 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:33 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:33 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae54b88174ed6b07bb616d232ccf2b2 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 7b3e18edb3c477ec82d275551c29a6e9 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/764f0a9f-2b1f-42f0-b213-346228f4cf2d_calibrate_871042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/764f0a9f-2b1f-42f0-b213-346228f4cf2d_calibrate_871042_36.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bdd780a6-9c4d-40e4-8903-56805ccda931_characterizeImage_871042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/bdd780a6-9c4d-40e4-8903-56805ccda931_characterizeImage_871042_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:34 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:35 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:35 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:36 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:36 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:37 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:37 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:38 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:38 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:39 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:40 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:40 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:41 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:41 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:42 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:43 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:43 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:44 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:44 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:45 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:45 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82de727c6f4a8907d3afa28992131205 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 80b1f331745747229d69edb5b8d9e944 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f43bacfa-741d-4c9d-931d-cacbd4dda7bb_calibrate_871043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f43bacfa-741d-4c9d-931d-cacbd4dda7bb_calibrate_871043_44.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a458bda8-e7ac-43ad-98d6-0a3d5db6dfd0_characterizeImage_871043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a458bda8-e7ac-43ad-98d6-0a3d5db6dfd0_characterizeImage_871043_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:46 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:47 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:47 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:48 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80e6ba06e6aa659787723e923644335f with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash d9a785dfc044b463482f117da9bb62c9 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:48 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86ab82e0-2a53-47ff-b267-259bbf8011fb_calibrate_871041_37.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86ab82e0-2a53-47ff-b267-259bbf8011fb_calibrate_871041_37.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/817bded6-aa6b-4d13-9dd7-cfcee23b8d76_characterizeImage_871041_37.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/817bded6-aa6b-4d13-9dd7-cfcee23b8d76_characterizeImage_871041_37.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87d677c79bfb6d240ac5355fbf35cb2a with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash bb63444b4992dfc9fffc0c033c0dea80 parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/481f030f-0fa4-473e-b05f-b544022ca06f_calibrate_871041_7.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/481f030f-0fa4-473e-b05f-b544022ca06f_calibrate_871041_7.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f5798468-4806-435c-9e0a-d103e72716be_characterizeImage_871041_7.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/f5798468-4806-435c-9e0a-d103e72716be_characterizeImage_871041_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:51 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:52 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:52 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 339ca029eac14ff1a485b3630b52e886 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 9ec60876d84b57f5923e639c74a42ff6 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0c992e25-7927-4aba-938d-e3f7276e8877_calibrate_871043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0c992e25-7927-4aba-938d-e3f7276e8877_calibrate_871043_54.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/050e19fb-2e9b-4a1d-b39e-7d0e4dbda75b_characterizeImage_871043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/050e19fb-2e9b-4a1d-b39e-7d0e4dbda75b_characterizeImage_871043_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:53 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:53 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:36:54 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:36:54 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:55 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:55 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:56 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:58 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:36:59 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f6a69cdb0b00dca7660ae2fc4ce42bd with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash e6bc43d84d2a609f3f471ba2567a9d3d parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7b8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/06d9681e-e8bd-42df-8954-06ed15438538_calibrate_871042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/06d9681e-e8bd-42df-8954-06ed15438538_calibrate_871042_25.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd3cb64f-f87c-49fe-90a8-3559d1bf7823_characterizeImage_871042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd3cb64f-f87c-49fe-90a8-3559d1bf7823_characterizeImage_871042_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:00 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:01 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:01 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:02 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:02 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:03 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:03 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87551ecde2a259ace38fd9afd9916ef1 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 103c6e6099326708470ad2483035dd5a parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7baca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca95ee70-eccf-4eb8-b5f2-2eb506341509_calibrate_871041_12.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca95ee70-eccf-4eb8-b5f2-2eb506341509_calibrate_871041_12.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/1f77c189-40b9-4d8e-916d-6bd2e27776bc_characterizeImage_871041_12.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/1f77c189-40b9-4d8e-916d-6bd2e27776bc_characterizeImage_871041_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:04 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:05 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:05 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:06 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:06 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f629d53a88ac00deac5b8956266d7869 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash a3ceca32639511b1aa0f69cd89e86e0a parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e8d348e0-48e1-41a9-bb80-79797105bd0d_calibrate_871041_1.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e8d348e0-48e1-41a9-bb80-79797105bd0d_calibrate_871041_1.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ac9dcad2-3ad4-44f0-ac09-e6a15eaed759_characterizeImage_871041_1.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ac9dcad2-3ad4-44f0-ac09-e6a15eaed759_characterizeImage_871041_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:07 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:07 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:08 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:09 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f48f298e18fd659cb2617f1a3a9e3a1 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash f35e3e1e544cb825cfd5b9948edd5d12 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb94eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/804a8201-4c46-4dec-86d2-146b6b3f30bb_calibrate_871043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/804a8201-4c46-4dec-86d2-146b6b3f30bb_calibrate_871043_3.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/824b966f-d852-46cb-88ec-e63c90b1e459_characterizeImage_871043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/824b966f-d852-46cb-88ec-e63c90b1e459_characterizeImage_871043_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:09 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:10 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:10 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:11 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:11 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:12 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:13 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:13 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:14 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:14 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:15 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:15 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:16 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:17 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:17 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:18 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:18 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:19 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dd26243380ef90b058244b282a0c4b5 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash d101a83b28744662795b2729cb107eff parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/25b89215-758d-4b58-95a3-fa991e6cb2dd_calibrate_871041_50.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/25b89215-758d-4b58-95a3-fa991e6cb2dd_calibrate_871041_50.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/10532b4c-0dc5-4752-ac66-9b06607521ea_characterizeImage_871041_50.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/10532b4c-0dc5-4752-ac66-9b06607521ea_characterizeImage_871041_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:20 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:20 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:21 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:21 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:22 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:23 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07017ab3a8e78d4357b53d2cd0f46e75 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 159a657dd1b9e77a3370fd3228abc012 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/549e0e36-14b9-4126-a12d-a1e74ed62989_calibrate_871042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/549e0e36-14b9-4126-a12d-a1e74ed62989_calibrate_871042_29.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3d9f017c-b96e-4ee2-bcfc-814c0caed233_characterizeImage_871042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3d9f017c-b96e-4ee2-bcfc-814c0caed233_characterizeImage_871042_29.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c43baeadeb695ce08684b854b5464828 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 6c1359c374cdb20d0e1a1722f97cf8cf parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b8cc16dc-d453-4661-ad66-2f473cd21d22_calibrate_871043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b8cc16dc-d453-4661-ad66-2f473cd21d22_calibrate_871043_23.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9ffc85c5-13f6-4128-a9c0-ac1ccca7b91c_characterizeImage_871043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/9ffc85c5-13f6-4128-a9c0-ac1ccca7b91c_characterizeImage_871043_23.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51a503a26d4549124eb1c685ca624441 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash bff46e037ae70e3cc3d623028c3827ff parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/16c883bf-e955-41c9-98ba-abf0619d2a2b_calibrate_871043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/16c883bf-e955-41c9-98ba-abf0619d2a2b_calibrate_871043_14.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2e6b9180-7aa8-499c-b878-f8fd2d68fd5a_characterizeImage_871043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/2e6b9180-7aa8-499c-b878-f8fd2d68fd5a_characterizeImage_871043_14.stderr 2024-03-10 15:37:28 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:37:29 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:29 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 1 failed parsl.dataflow.dflow ERROR: Task 10 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f2ae214773aad8dd10cd776b3005d9f7 with result from task 10 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 11 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 11 try 0 failed parsl.dataflow.dflow INFO: Task 11 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/e7631997-bb8a-4053-8309-12dae98829d8_calibrate_871043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/e7631997-bb8a-4053-8309-12dae98829d8_calibrate_871043_2.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/f7a6c831-2cd1-4dd5-864a-820622d85463_characterizeImage_871043_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:31 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:32 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:33 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:34 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:34 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:35 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:35 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:36 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:37 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:37 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:38 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:38 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed01baaf684b1a2d205ddd26a9956f9 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 0addd09c259ac3d7ec481a18017cc445 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7480f460-a43f-479b-ba1e-7b1f77722450_calibrate_871041_11.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7480f460-a43f-479b-ba1e-7b1f77722450_calibrate_871041_11.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbc049de-7269-42a3-89ed-134bc72a494f_characterizeImage_871041_11.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/bbc049de-7269-42a3-89ed-134bc72a494f_characterizeImage_871041_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:39 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:39 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:40 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:41 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:41 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:42 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:42 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:43 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:43 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:44 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:45 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:45 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:46 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:46 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:47 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:47 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:48 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:49 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:49 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:50 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:50 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:51 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee1ab2e83fdb28f10153877d2260be23 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash e601814a59aaa9ad5c51dabb25edf852 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/70160937-4461-4605-87d7-cc07ebeb5db4_calibrate_871043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/70160937-4461-4605-87d7-cc07ebeb5db4_calibrate_871043_12.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7934e613-f53e-4dbe-b4f1-190ac9c748df_characterizeImage_871043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7934e613-f53e-4dbe-b4f1-190ac9c748df_characterizeImage_871043_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:51 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:51 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2179d40dcd784683db9981e77f1a1f9 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash b72d4d880f9062fb9d13bacc4778bb36 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6dfa7ac8-733a-4372-8312-d872496d6f22_calibrate_871043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6dfa7ac8-733a-4372-8312-d872496d6f22_calibrate_871043_46.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/dc5b9aea-555a-48cc-966c-97c2672e378e_characterizeImage_871043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/dc5b9aea-555a-48cc-966c-97c2672e378e_characterizeImage_871043_46.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 226df7d44b3787a2e7bd1e17a7f559ce with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 45406deb8f7f42c37397f627a84a415a parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7bafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/45f9e681-49eb-4a10-a848-657723d6f25a_calibrate_871042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/45f9e681-49eb-4a10-a848-657723d6f25a_calibrate_871042_60.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/142d70f5-4761-466b-bc37-d8fc184ccc91_characterizeImage_871042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/142d70f5-4761-466b-bc37-d8fc184ccc91_characterizeImage_871042_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:52 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:52 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:53 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:54 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a141f8369fd6f23da469470c2af13529 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 9a7f107f9fe1fc87ba3afc8d44da6aff parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7ba7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9018e033-f706-4c78-8f68-1f1d44760d45_calibrate_871042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9018e033-f706-4c78-8f68-1f1d44760d45_calibrate_871042_37.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/0ccd6633-0176-4bf9-b7cc-a12800519856_characterizeImage_871042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/0ccd6633-0176-4bf9-b7cc-a12800519856_characterizeImage_871042_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:54 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:55 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:55 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:37:56 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 185 parsl.dataflow.strategy DEBUG: Executor multi has 185 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 185 parsl.dataflow.strategy DEBUG: Executor multi has 185 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:38:01 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:38:02 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:02 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:04 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:05 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:06 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:07 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:07 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:08 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:08 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:09 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:09 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:10 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:11 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:11 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:12 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:12 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:13 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:14 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:14 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:15 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:15 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:16 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6268a13ea9a206dd7a4762d14a3922da with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 4b12afe045ce354e3c921feeaf53f385 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca92000c-e43c-4ecc-8eb1-1a8b50f621bf_calibrate_871041_20.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca92000c-e43c-4ecc-8eb1-1a8b50f621bf_calibrate_871041_20.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/686d9bf8-fc9b-4dab-9d76-bbcf06e00e84_characterizeImage_871041_20.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/686d9bf8-fc9b-4dab-9d76-bbcf06e00e84_characterizeImage_871041_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:16 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae8d87d725a2e2efd45dff7fb77457c7 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 3a7d5590efe6b41894d615c8aad9341e parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b783c40f-39f1-4aff-84dc-97995af08de9_calibrate_871041_47.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b783c40f-39f1-4aff-84dc-97995af08de9_calibrate_871041_47.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/47ff830c-3a58-4604-b972-e2e523de2b31_characterizeImage_871041_47.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/47ff830c-3a58-4604-b972-e2e523de2b31_characterizeImage_871041_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:17 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:17 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:18 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:19 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:19 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:20 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb91a2cbcdbcfb18842003ae889cb368 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 2b4617d62743f423bf1c5285e77a1860 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8431a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/285a39b1-6d72-4077-a144-2adeedd946f2_calibrate_871041_62.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/285a39b1-6d72-4077-a144-2adeedd946f2_calibrate_871041_62.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b2071292-3a92-4796-afbb-0094873bbf83_characterizeImage_871041_62.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/b2071292-3a92-4796-afbb-0094873bbf83_characterizeImage_871041_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:20 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9d6dbadd6b8321db28ad320bdabd269 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 518e7978134620b63a755b45d371c935 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fbabdbb8-faaa-48e2-89d8-7771910e38e4_calibrate_871042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fbabdbb8-faaa-48e2-89d8-7771910e38e4_calibrate_871042_10.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/77245805-c265-4953-b997-b8873c5da87b_characterizeImage_871042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/77245805-c265-4953-b997-b8873c5da87b_characterizeImage_871042_10.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9128b761b880b992f7fad75155c45ba0 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash b5029d5db7b708267f0a92ba477a37be parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca841760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/23886196-662d-491d-9950-76f1fb94d94b_calibrate_871041_45.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/23886196-662d-491d-9950-76f1fb94d94b_calibrate_871041_45.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8bbf40b9-0074-43f7-9c7a-2fde0b5b2dbe_characterizeImage_871041_45.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/8bbf40b9-0074-43f7-9c7a-2fde0b5b2dbe_characterizeImage_871041_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:21 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:22 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:22 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6697a31d1759f10eed5762a6e4087754 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 8dbe8df763fd64cfa977a096c439315a parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/afcc7f0c-d44e-446e-9021-a378f95c6513_calibrate_871041_40.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/afcc7f0c-d44e-446e-9021-a378f95c6513_calibrate_871041_40.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9b592ef9-1850-4693-bbfc-c4bee8de143b_characterizeImage_871041_40.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/9b592ef9-1850-4693-bbfc-c4bee8de143b_characterizeImage_871041_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:23 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61cf3e3b27b78d3b1ee6347b2d5c2ff8 with result from task 421 parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:23 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 67a8cf9495585d56e1345edcd09c7ba0 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/46f47ab6-3e61-4261-97ab-9805a7c193f2_calibrate_871042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/46f47ab6-3e61-4261-97ab-9805a7c193f2_calibrate_871042_19.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ffb4b512-adaa-4b33-9f4c-ea5f675c305f_characterizeImage_871042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ffb4b512-adaa-4b33-9f4c-ea5f675c305f_characterizeImage_871042_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:24 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:25 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:25 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:26 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:26 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:27 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:27 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:28 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:28 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:29 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ebc74e33e50dba9f95666d8e0dd883 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 4287bde667eaa509d8e891aa063ac232 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5eedec86-a83a-40b4-ab78-466c07b2942f_calibrate_871043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5eedec86-a83a-40b4-ab78-466c07b2942f_calibrate_871043_17.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/43cec55c-5016-450b-876e-602ba678c64d_characterizeImage_871043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/43cec55c-5016-450b-876e-602ba678c64d_characterizeImage_871043_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:30 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 185 parsl.dataflow.strategy DEBUG: Executor multi has 185 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 185 parsl.dataflow.strategy DEBUG: Executor multi has 185 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b62c5459f459ee1a13c64922dc5d02a with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash aa12cd501c7f521004e2876985c429ff parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8420c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d35c6298-eadf-467c-ba68-61d386de726c_calibrate_871043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d35c6298-eadf-467c-ba68-61d386de726c_calibrate_871043_19.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/82a8a4ab-b7d9-4b09-9d0e-c920788a3cb7_characterizeImage_871043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/82a8a4ab-b7d9-4b09-9d0e-c920788a3cb7_characterizeImage_871043_19.stderr 2024-03-10 15:38:35 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:38:35 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d64f11b215ec15ef869746955852cb5 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 6a26e5bec23a38d769ae9c6b234d219c parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/09e6ef83-7b84-4615-95d2-2942d85dbced_calibrate_871041_58.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/09e6ef83-7b84-4615-95d2-2942d85dbced_calibrate_871041_58.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7e07be96-038a-43d3-9bf3-78c091ad60b5_characterizeImage_871041_58.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7e07be96-038a-43d3-9bf3-78c091ad60b5_characterizeImage_871041_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:37 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:38 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:38 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:39 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:39 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:40 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:41 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:41 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8112a2aa8f7d67d0f13c80548af0861b with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 562495ee244660ccf4ad17aa465ed16f parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a4821404-9db9-4560-b34b-b8f872aa5efc_calibrate_871042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a4821404-9db9-4560-b34b-b8f872aa5efc_calibrate_871042_18.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7cb1390f-a607-4a5b-b970-550bf96e3987_characterizeImage_871042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7cb1390f-a607-4a5b-b970-550bf96e3987_characterizeImage_871042_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:42 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64738c953a6384b59834102bb3b23a36 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash d06f8dc641b3d44f7b844e7af4bc48c4 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/9e9719aa-1497-4cad-b511-e678c32c7340_calibrate_871041_14.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/9e9719aa-1497-4cad-b511-e678c32c7340_calibrate_871041_14.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/115754c0-7dc1-446f-8c0f-3ad26f23664c_characterizeImage_871041_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/115754c0-7dc1-446f-8c0f-3ad26f23664c_characterizeImage_871041_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:42 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:43 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:44 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:44 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:44 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:45 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:46 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:46 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:47 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:47 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44a43deaa399a1e10d7a7b0fa55453e4 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 14d071de99f0d969996dcdba9e5ec227 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1f04832-9be7-4e02-ade1-1a5f0fbb465c_calibrate_871042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1f04832-9be7-4e02-ade1-1a5f0fbb465c_calibrate_871042_44.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ad3afc09-7be3-4142-ac02-77ce747b10be_characterizeImage_871042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/ad3afc09-7be3-4142-ac02-77ce747b10be_characterizeImage_871042_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:48 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:48 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:49 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:50 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:50 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:51 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:51 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:52 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:53 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b3b25aea87e317e3e62ed2848a899c9 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5b8b1b1d-3352-4bed-b05f-90ad5af34f0e_calibrate_871041_31.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5b8b1b1d-3352-4bed-b05f-90ad5af34f0e_calibrate_871041_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:53 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:54 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48e7cf4fde0b2c1cf1a30a9eb7d4b5fb with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 4593ecc7614599f5710bb5097d51216d parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8a327a39-1ce4-4436-a89e-796769832460_calibrate_871041_39.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8a327a39-1ce4-4436-a89e-796769832460_calibrate_871041_39.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7cabfa77-9f9f-4dc7-8d33-f1b1854793b8_characterizeImage_871041_39.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7cabfa77-9f9f-4dc7-8d33-f1b1854793b8_characterizeImage_871041_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:54 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:55 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:55 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:56 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:57 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:57 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:58 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:58 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:38:59 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:00 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:00 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:01 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53cc552bae30fd1062520afb77c63bf3 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 29e636d5ccccde35dbcb2bdb6e78e3c5 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e2f0ec17-25be-4c68-b357-f73b3d34d20d_calibrate_871042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e2f0ec17-25be-4c68-b357-f73b3d34d20d_calibrate_871042_22.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7536cb80-fc28-438f-bc94-da67946ccce5_characterizeImage_871042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/7536cb80-fc28-438f-bc94-da67946ccce5_characterizeImage_871042_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:01 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:02 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:03 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 184 parsl.dataflow.strategy DEBUG: Executor multi has 184 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 184 parsl.dataflow.strategy DEBUG: Executor multi has 184 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7be659a578dca3c2e47f7e9441704bf0 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash f5d8fc1c665609e437a4662c5f3307f2 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/cf7b59e5-7eef-4b52-b7e7-0da9e7dad0ed_calibrate_871041_27.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/cf7b59e5-7eef-4b52-b7e7-0da9e7dad0ed_calibrate_871041_27.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d4df33c8-0da9-4fa0-bf8b-1d3d60ff2225_characterizeImage_871041_27.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/d4df33c8-0da9-4fa0-bf8b-1d3d60ff2225_characterizeImage_871041_27.stderr 2024-03-10 15:39:08 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:39:08 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:10 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:11 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69cb1cecf1b1ff632103aa2c9fe4d511 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 9307779c7d5c2ca7862639d6166ef1e2 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/41e24fa1-3c9a-4afe-8ebb-6f952b0e0354_calibrate_871041_41.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/41e24fa1-3c9a-4afe-8ebb-6f952b0e0354_calibrate_871041_41.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/75749a2d-01e8-43d5-b096-c8654cc3cf56_characterizeImage_871041_41.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/75749a2d-01e8-43d5-b096-c8654cc3cf56_characterizeImage_871041_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:11 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:12 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:12 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:13 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:13 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b9efdb3e8e539f7c44c9ecaac97eae6 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 1dc27b177e05ef0fcd0ebd9fc1a19165 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7fb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/84094d57-b4e5-45d3-8e25-b231ac2e2596_calibrate_871043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/84094d57-b4e5-45d3-8e25-b231ac2e2596_calibrate_871043_60.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ac7b3a7e-c87f-41c5-b206-d238cf43bf41_characterizeImage_871043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/ac7b3a7e-c87f-41c5-b206-d238cf43bf41_characterizeImage_871043_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:14 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:14 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:15 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:16 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 519d9b4bf9c7fe6340188be8bbc0eb6f with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 9a7ec758bd2bdf277eb7a91fc7d70274 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/815b9560-b659-49cc-b55e-036d9cc559fb_calibrate_871043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/815b9560-b659-49cc-b55e-036d9cc559fb_calibrate_871043_16.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5c01169b-d2e5-42bf-9dd7-e634f449d462_characterizeImage_871043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/5c01169b-d2e5-42bf-9dd7-e634f449d462_characterizeImage_871043_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:16 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:17 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:17 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23015865bf3b250c79cf6f1347a687f1 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/bd5016c7-fc40-444f-9ea1-69044b727b57_calibrate_871042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/bd5016c7-fc40-444f-9ea1-69044b727b57_calibrate_871042_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:18 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec04b30db85ac020eceb1b6e197aab7e with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f8a4ab6-6beb-4bb5-bd43-93fa5cf392d7_calibrate_871042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f8a4ab6-6beb-4bb5-bd43-93fa5cf392d7_calibrate_871042_31.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8472b65837b3fc036afdd4847b51e180 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash f5f2a2c406335cfa26289e86626c8a1d parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c846677a-3f74-46f3-8b35-5133b515005c_calibrate_871042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c846677a-3f74-46f3-8b35-5133b515005c_calibrate_871042_43.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d1acd1de-e32a-4c31-ace3-7907acffa50c_characterizeImage_871042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/d1acd1de-e32a-4c31-ace3-7907acffa50c_characterizeImage_871042_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:19 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b98f5b9e50b3165ae33ffed631deb501 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash c3b6413484c1c753aa5601fbde5dd4ef parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/8d867a21-3b26-4e37-8592-316657932738_calibrate_871043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/8d867a21-3b26-4e37-8592-316657932738_calibrate_871043_55.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/8391be91-4bdf-4380-8640-80068c62c53b_characterizeImage_871043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/8391be91-4bdf-4380-8640-80068c62c53b_characterizeImage_871043_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:19 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:20 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:20 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:21 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:22 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:22 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:23 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:23 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:24 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc1b01a0b5b8009836908b4681d3a9e1 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash f569ee2b9c6dd888e4e1cba269e61ae1 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b7143b31-f45d-48a4-9401-d18a0f197da0_calibrate_871041_17.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b7143b31-f45d-48a4-9401-d18a0f197da0_calibrate_871041_17.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6e82df33-d544-4eb3-886a-1ce4c96ec65d_characterizeImage_871041_17.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/6e82df33-d544-4eb3-886a-1ce4c96ec65d_characterizeImage_871041_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:25 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:25 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2a57e0cf0643928009baa3980b5f845 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/508b45c0-f8cb-4e5d-a5fa-aa3810206caa_calibrate_871041_55.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/508b45c0-f8cb-4e5d-a5fa-aa3810206caa_calibrate_871041_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:26 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7d275338d71fa0f99693db27fcf919c with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4016b4c3-036f-4994-aa31-99d241708cc8_calibrate_871041_54.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4016b4c3-036f-4994-aa31-99d241708cc8_calibrate_871041_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:26 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:27 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:27 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:28 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:29 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9023e411580f840bad7099f5657689d with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 0a032eda2c16c94cff3dabd65584a2f6 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca843b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ea0786a9-09d3-49e9-b7e0-74928ec14a49_calibrate_871042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ea0786a9-09d3-49e9-b7e0-74928ec14a49_calibrate_871042_41.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5a4ba913-11e5-4360-a2c6-f9944719340f_characterizeImage_871042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5a4ba913-11e5-4360-a2c6-f9944719340f_characterizeImage_871042_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:29 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:30 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed573b51004e97b744989849e5f595b3 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash c0e5c7a0ea0142b2b3adebba74dcebda parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/cafe4761-0c96-4843-b346-cf0a5842b114_calibrate_871043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/cafe4761-0c96-4843-b346-cf0a5842b114_calibrate_871043_24.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c90f0b9d-6748-4495-9637-45c0c9d52111_characterizeImage_871043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c90f0b9d-6748-4495-9637-45c0c9d52111_characterizeImage_871043_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:30 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:31 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:31 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:32 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:33 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:33 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:34 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:34 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:35 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:35 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 180 parsl.dataflow.strategy DEBUG: Executor multi has 180 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 180 parsl.dataflow.strategy DEBUG: Executor multi has 180 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64b5747e887caee9ed0b9cdf51ac2699 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94be30f3-e7f2-46d3-a5f7-21783b4dc34d_calibrate_871043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94be30f3-e7f2-46d3-a5f7-21783b4dc34d_calibrate_871043_49.stderr 2024-03-10 15:39:41 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:39:41 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:42 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:42 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:43 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:44 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b2a818fc20ffe7585ea839e1e24388c with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 3e11e6bc22592734d92aa6be0bf0485e parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b188633-eb0b-4f61-914d-e69808817f6d_calibrate_871041_15.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b188633-eb0b-4f61-914d-e69808817f6d_calibrate_871041_15.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/32ae3f27-4f3e-4240-971b-0634665a8d21_characterizeImage_871041_15.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/32ae3f27-4f3e-4240-971b-0634665a8d21_characterizeImage_871041_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:44 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:45 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:45 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:46 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:47 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:47 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c97a993d06ab51580ab850ca3d11018 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash a80efc366981764de06c2d5c4d6dac5d parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/23942b39-d3f8-47ba-95ea-082264867b2f_calibrate_871043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/23942b39-d3f8-47ba-95ea-082264867b2f_calibrate_871043_18.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/55ede27d-4502-4c76-9390-e567a4400db3_characterizeImage_871043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/55ede27d-4502-4c76-9390-e567a4400db3_characterizeImage_871043_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:48 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:48 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:49 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:49 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b58a2367f6995ea975a73fe0535326bb with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 4395cd8bb081c33254ffd810eb6990cd parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/076557d5-37bd-4912-8454-44b9e833a7a9_calibrate_871043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/076557d5-37bd-4912-8454-44b9e833a7a9_calibrate_871043_35.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3f460ad6-1c2d-4ea3-8c04-e5bc38a29508_characterizeImage_871043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/3f460ad6-1c2d-4ea3-8c04-e5bc38a29508_characterizeImage_871043_35.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e56ad504969dc9e7a2a6a664baf9245 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/eaead91d-6044-4c16-83ed-e2c0c18482ae_calibrate_871042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/eaead91d-6044-4c16-83ed-e2c0c18482ae_calibrate_871042_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:50 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:51 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:51 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3a434ed5881d9d73ca609a127277d0d with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 965876de713349b323c9f85c66abe688 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d555e65a-d460-43db-8bd0-320eb497c54c_calibrate_871041_51.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d555e65a-d460-43db-8bd0-320eb497c54c_calibrate_871041_51.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/e96808ae-2a62-4cf1-ae2d-3d9e1ec55374_characterizeImage_871041_51.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/e96808ae-2a62-4cf1-ae2d-3d9e1ec55374_characterizeImage_871041_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:52 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:52 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:53 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:54 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:54 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:55 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:55 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:56 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9a4f0b62850c33ae6ab302edbeb0631 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6c54fc22-0de5-491e-a98e-528f0d58ad20_calibrate_871043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6c54fc22-0de5-491e-a98e-528f0d58ad20_calibrate_871043_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 262 try 1 failed parsl.dataflow.dflow ERROR: Task 262 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 76938ba7a791a7eabdd13e8914335187 with result from task 262 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 263 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 263 try 0 failed parsl.dataflow.dflow INFO: Task 263 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/2a8d4dd0-c7f6-4c5c-85e0-fbc12f1ae362_calibrate_871041_2.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/2a8d4dd0-c7f6-4c5c-85e0-fbc12f1ae362_calibrate_871041_2.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/ae9b6ea3-f39f-4633-9dfd-a6fbd94f9972_characterizeImage_871041_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:56 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:57 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f15fd0389ea983c14892581791c5c4b with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ac965867-5a63-459b-849f-fda2baeafe55_calibrate_871042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ac965867-5a63-459b-849f-fda2baeafe55_calibrate_871042_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:58 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:58 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:39:59 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:00 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea77e2813df827e24ec4ba853c007a64 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 4dc5a44fa98292af08e7251a7ce3b964 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0d3879af-d879-4ff1-8262-540e193639a2_calibrate_871043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0d3879af-d879-4ff1-8262-540e193639a2_calibrate_871043_20.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b193aae3-4570-4fa0-a8eb-dba40771c7e0_characterizeImage_871043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b193aae3-4570-4fa0-a8eb-dba40771c7e0_characterizeImage_871043_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:00 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:01 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:01 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:02 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:02 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:03 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:04 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:04 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c0d4d616374cf7b0b0c2402d99efb90 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/52b8465d-a149-4ea3-86a8-ba3e805cd891_calibrate_871041_57.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/52b8465d-a149-4ea3-86a8-ba3e805cd891_calibrate_871041_57.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1754dab16e5d9a9fcf0cc1315eda3a30 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/643f0dda-1535-4da2-8cae-77584432908d_calibrate_871043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/643f0dda-1535-4da2-8cae-77584432908d_calibrate_871043_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:05 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:05 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:06 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5389c62fcd04689bec5af47ecb624bc4 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash c9ae35e7134c644768db12a20c394576 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6ab4ed3b-6743-4694-8733-6936beaf7ef4_calibrate_871041_9.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6ab4ed3b-6743-4694-8733-6936beaf7ef4_calibrate_871041_9.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/66cefeb5-6f0f-4e50-8f10-64d97e25f197_characterizeImage_871041_9.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/66cefeb5-6f0f-4e50-8f10-64d97e25f197_characterizeImage_871041_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:06 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:07 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:08 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f58450cbc3ca010d843f0c6fa8fd5ccd with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2e616136-0832-4cd1-b897-1b16b219222d_calibrate_871042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2e616136-0832-4cd1-b897-1b16b219222d_calibrate_871042_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 172 parsl.dataflow.strategy DEBUG: Executor multi has 172 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 172 parsl.dataflow.strategy DEBUG: Executor multi has 172 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18f0eea5d89b85e8aa008625c9cfb913 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/37c53fa8-45a9-40b3-947b-b8227e54a00c_calibrate_871043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/37c53fa8-45a9-40b3-947b-b8227e54a00c_calibrate_871043_57.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9e8fdbdf6189b4f8f349d1a7541b506 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash bde20254238507a562f5f7b9955b5a14 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d8586ef9-1140-4091-b2a3-369395283cb6_calibrate_871043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d8586ef9-1140-4091-b2a3-369395283cb6_calibrate_871043_28.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/0886dbae-636c-4efc-9ec7-70c3be3ac70f_characterizeImage_871043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/0886dbae-636c-4efc-9ec7-70c3be3ac70f_characterizeImage_871043_28.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99b06d8f224c308ca116f339f8ae30a9 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 70f25f85ee0b2a8d0e2538eb6c2944aa parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca8437e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0fd99b48-a4cf-4b44-920a-598c399b47cd_calibrate_871043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0fd99b48-a4cf-4b44-920a-598c399b47cd_calibrate_871043_32.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c92c56f7-791f-4b1e-a97f-e3b4bc327c49_characterizeImage_871043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/c92c56f7-791f-4b1e-a97f-e3b4bc327c49_characterizeImage_871043_32.stderr 2024-03-10 15:40:13 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:40:13 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:14 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dac40aaec0de89bae5125d6d2e71f03 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/26df321c-1535-48fc-87a0-1b485131237e_calibrate_871043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/26df321c-1535-48fc-87a0-1b485131237e_calibrate_871043_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:15 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:15 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:16 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:16 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d70f61e3ddb120acd6c23040395e88 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4352abed-ac8a-47f4-9cee-8fcf4614c5ac_calibrate_871041_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/4352abed-ac8a-47f4-9cee-8fcf4614c5ac_calibrate_871041_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:17 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:18 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3b503da1a4eb8818ee4cdec096b16f6 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/220087c0-2650-446d-93a9-69480679784b_calibrate_871041_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/220087c0-2650-446d-93a9-69480679784b_calibrate_871041_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:18 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:19 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:19 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:20 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dba972fc3521a512d507336845bba9f with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash d8a800e59254ba3ae276b0f9ef02d652 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/11e43bcb-57ac-4f09-a723-4c27a7a76a57_calibrate_871043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/11e43bcb-57ac-4f09-a723-4c27a7a76a57_calibrate_871043_8.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e95a6277-0bd4-4ee7-a392-e9966882389e_characterizeImage_871043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/e95a6277-0bd4-4ee7-a392-e9966882389e_characterizeImage_871043_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:21 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:21 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cd2f70acf390f48ed761e15c4a79f09 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 70c8bd816973b36ddcb9316d4ac43f7f parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a59368f8-1733-4298-85a7-dcb898f76279_calibrate_871042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a59368f8-1733-4298-85a7-dcb898f76279_calibrate_871042_49.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/93783b46-1ad3-4cad-b5b0-91dd149eb7bb_characterizeImage_871042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/93783b46-1ad3-4cad-b5b0-91dd149eb7bb_characterizeImage_871042_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:22 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:22 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3d3969e1ba59672a941bded44f94e9d with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash d92ab6cd69bccac82a619d9947aa8609 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb8e1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c610941a-9283-4dc5-adbc-2f5fd5f1ab01_calibrate_871041_13.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c610941a-9283-4dc5-adbc-2f5fd5f1ab01_calibrate_871041_13.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7744b24c-9095-4bac-935e-fa9e7ba3da6a_characterizeImage_871041_13.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/7744b24c-9095-4bac-935e-fa9e7ba3da6a_characterizeImage_871041_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:23 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6b9cef09205935b145e03dc1c856ee0 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fc5ae9aa-e42e-4202-9e66-5537da169d58_calibrate_871042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fc5ae9aa-e42e-4202-9e66-5537da169d58_calibrate_871042_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:23 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:24 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:24 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67fdc8749e041f55eed45761d7c4c430 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/94763d12-4155-45b5-9e04-5eb5b0513987_calibrate_871042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/94763d12-4155-45b5-9e04-5eb5b0513987_calibrate_871042_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:25 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:25 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:26 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:27 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:27 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:28 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:28 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab831abac9ac73e1c77a96827ffd27d7 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/45947ebc-724d-48c5-a041-5487bf7cc78b_calibrate_871043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/45947ebc-724d-48c5-a041-5487bf7cc78b_calibrate_871043_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:29 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:29 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:30 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:30 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd65d2af28f611a95ea33f56d167fd08 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3af544fe-ac85-436a-a9fa-ff720b1f1af6_calibrate_871042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3af544fe-ac85-436a-a9fa-ff720b1f1af6_calibrate_871042_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:31 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:31 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:32 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:32 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:33 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:34 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:34 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:35 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:35 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf9b4fc87c1bb90542edbbd4f11f3ff5 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1703b98f-033a-4cbe-bd7b-670da71b5e8c_calibrate_871042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1703b98f-033a-4cbe-bd7b-670da71b5e8c_calibrate_871042_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:36 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34b9a390bb8068277dc8e93a12340029 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/77555292-57b6-47a2-a482-06e534e14c31_calibrate_871043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/77555292-57b6-47a2-a482-06e534e14c31_calibrate_871043_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:37 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 844310eb168ede173d904c192b9dd155 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5cffb1e2-a9d7-4e63-8ae6-e122bc612545_calibrate_871042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5cffb1e2-a9d7-4e63-8ae6-e122bc612545_calibrate_871042_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:37 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96a444163c01f529f2759d445b81abd6 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 394c8e686c914c4335d93514094c08e5 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0f4836ca-971c-437f-94de-1ccd8a833d98_calibrate_871043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0f4836ca-971c-437f-94de-1ccd8a833d98_calibrate_871043_1.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/676fe280-0676-4a2e-b00b-83410bef04f8_characterizeImage_871043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/676fe280-0676-4a2e-b00b-83410bef04f8_characterizeImage_871043_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:38 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3de3998993fd8fc932a5b1d77085942b with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/54b44876-67a0-488a-9bc8-4f7341d74a11_calibrate_871042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/54b44876-67a0-488a-9bc8-4f7341d74a11_calibrate_871042_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:38 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:39 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 160 parsl.dataflow.strategy DEBUG: Executor multi has 160 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 160 parsl.dataflow.strategy DEBUG: Executor multi has 160 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7078c25f27a5469931af404430e6aa19 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/301518ca-bf5d-47d3-9cad-a7f0fba99a36_calibrate_871041_26.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/301518ca-bf5d-47d3-9cad-a7f0fba99a36_calibrate_871041_26.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e62a04ac9724318b5bfdf800dcf7fcb with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e6022611-33dc-4d9b-9d52-f8e12503637e_calibrate_871041_5.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e6022611-33dc-4d9b-9d52-f8e12503637e_calibrate_871041_5.stderr 2024-03-10 15:40:44 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:40:45 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:45 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:46 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:46 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:47 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:48 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6effe58fb603608714845eedcdfbd49c with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5c62c41f-868c-421c-9ff8-4be6af5a739a_calibrate_871043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5c62c41f-868c-421c-9ff8-4be6af5a739a_calibrate_871043_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:48 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:49 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:49 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:50 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:50 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 881c6f1c83698c7b7049ca6f001e4821 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash ae4c9b40a3128f8d4a585845d34a6d7f parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/12d8a490-25ca-4545-a09a-a80210dc14bc_calibrate_871043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/12d8a490-25ca-4545-a09a-a80210dc14bc_calibrate_871043_36.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7b7b22d1-5757-48f0-a73e-c7281bdf61db_characterizeImage_871043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/7b7b22d1-5757-48f0-a73e-c7281bdf61db_characterizeImage_871043_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:51 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2156c66be10418e7bd21f4ba5b5b47ca with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1317fc37-3cf3-4950-8589-9ff705336e32_calibrate_871043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1317fc37-3cf3-4950-8589-9ff705336e32_calibrate_871043_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:52 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:52 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:53 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3af804e2fd021a424da9665bf78d29c with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5847f6cd-7c67-40cb-82b3-cc77e1b3fc71_calibrate_871042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/5847f6cd-7c67-40cb-82b3-cc77e1b3fc71_calibrate_871042_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:53 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 870f07df776e08d718671f8f9cff06df with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a1ed9a5b-994c-41cc-9492-5da61687d757_calibrate_871043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a1ed9a5b-994c-41cc-9492-5da61687d757_calibrate_871043_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:54 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:55 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:55 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:56 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ea7cd387b8f79ae348a366a1afd3ba with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a7c81b42-3b55-4e3c-b3fe-c18df9459d9e_calibrate_871041_30.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a7c81b42-3b55-4e3c-b3fe-c18df9459d9e_calibrate_871041_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:56 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:57 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:58 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:58 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d621f1527d1d4b7c7db6d617b556494 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2bba1968-0ad8-442c-9e0b-6cfbae816b14_calibrate_871042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2bba1968-0ad8-442c-9e0b-6cfbae816b14_calibrate_871042_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:59 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:40:59 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:00 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:01 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a821ee126bd85217056f02749b737079 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b644631c-ef82-41e2-9b1c-1d7151dd4e8b_calibrate_871041_19.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b644631c-ef82-41e2-9b1c-1d7151dd4e8b_calibrate_871041_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:01 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05b1a8ca609f399fe1cacf8dd3c9b206 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0702634-ffef-4839-a016-ae7810877a7d_calibrate_871042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0702634-ffef-4839-a016-ae7810877a7d_calibrate_871042_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:02 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:02 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:03 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:04 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:04 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:05 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06c3e16447e0773d7a6e47dd936e4af2 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/dad9b5af-2d7c-4bd8-bbce-291c0fdc34d3_calibrate_871041_32.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/dad9b5af-2d7c-4bd8-bbce-291c0fdc34d3_calibrate_871041_32.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9edced820766b97bd051686d541529e9 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/56e5b3f0-1bde-4d6c-a1ec-ddb099436fc6_calibrate_871043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/56e5b3f0-1bde-4d6c-a1ec-ddb099436fc6_calibrate_871043_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:06 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca972714249dd0054a51420ce71142d4 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1c107be5-7222-41e6-981b-b93c63a8bf11_calibrate_871042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1c107be5-7222-41e6-981b-b93c63a8bf11_calibrate_871042_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:06 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08cc2c697daa6679ca3e36a66dc3df3d with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8dd113c0-3dcb-4e23-b7cc-5f406687132e_calibrate_871041_56.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8dd113c0-3dcb-4e23-b7cc-5f406687132e_calibrate_871041_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:07 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:07 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3bd63bb4be9fece383ecdf7b1f222f4 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/50c14087-758d-4a2e-981a-e7e484167652_calibrate_871042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/50c14087-758d-4a2e-981a-e7e484167652_calibrate_871042_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:08 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:09 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:09 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:10 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:10 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:11 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:11 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 145 parsl.dataflow.strategy DEBUG: Executor multi has 145 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 145 parsl.dataflow.strategy DEBUG: Executor multi has 145 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1830df902d8ae96b869db476a87b53 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f5036d46-9b74-4e8d-ba15-e70c1d83bc09_calibrate_871042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f5036d46-9b74-4e8d-ba15-e70c1d83bc09_calibrate_871042_53.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32151581b4a8475549075d33900f94a with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/60c0985f-3cba-43ee-b41e-b90b478d8d60_calibrate_871041_18.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/60c0985f-3cba-43ee-b41e-b90b478d8d60_calibrate_871041_18.stderr 2024-03-10 15:41:17 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:41:17 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:18 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:18 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:19 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:20 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 982d198a58dbbaa7a046d6d9ea0b5363 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 8095edb1ac6dcb386ece03e2ab3098fa parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3d969d05-5234-477e-959f-ec53e7b3eb06_calibrate_871043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3d969d05-5234-477e-959f-ec53e7b3eb06_calibrate_871043_56.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a64c908e-f4af-4254-931f-e55eca41c7a8_characterizeImage_871043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/a64c908e-f4af-4254-931f-e55eca41c7a8_characterizeImage_871043_56.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d662bdb7d93b89c790b550c4e96aec79 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/87239282-3f61-4332-82d7-437891f930f5_calibrate_871042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/87239282-3f61-4332-82d7-437891f930f5_calibrate_871042_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:20 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:21 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:21 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:22 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:23 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:23 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:24 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:24 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:25 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:26 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:26 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60830b6ffdceea72ec5aff5480425af3 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/08e94fd8-1d08-48a4-a0bd-caa49921aa6f_calibrate_871041_36.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/08e94fd8-1d08-48a4-a0bd-caa49921aa6f_calibrate_871041_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:27 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:27 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:28 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:29 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 526d012be05cb3fb083fcce3fe14e155 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5a0e4199-0b21-49bf-8969-9089c515539d_calibrate_871041_44.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5a0e4199-0b21-49bf-8969-9089c515539d_calibrate_871041_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:29 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:30 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:31 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 522291d61b64542056eb4bcad489e20d with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/88222bc3-7fd6-43f3-8d0e-ec44508c81fe_calibrate_871043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/88222bc3-7fd6-43f3-8d0e-ec44508c81fe_calibrate_871043_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:31 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:32 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cd51cc0f5c6baf675062fc53b27d46c with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 869f3d56e35ad431add425270ab61bef parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca840180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b678b4a9-ec0b-4aee-a3f9-e194a69a0095_calibrate_871042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b678b4a9-ec0b-4aee-a3f9-e194a69a0095_calibrate_871042_1.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9c0ab8a2-ae6a-4d27-8a1e-e26e76775734_characterizeImage_871042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/9c0ab8a2-ae6a-4d27-8a1e-e26e76775734_characterizeImage_871042_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:32 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:33 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fee91bc5e91d41a09637fa6934b1e9a with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/28e8b486-e782-4a1d-8faf-b81ee73ff0e2_calibrate_871043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/28e8b486-e782-4a1d-8faf-b81ee73ff0e2_calibrate_871043_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:34 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:34 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:35 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:36 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:36 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:37 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64e7a1d361eb292ecaa085fc4c6f213f with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/acc688c3-db11-4c6a-bfb4-7d16bf234521_calibrate_871041_43.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/acc688c3-db11-4c6a-bfb4-7d16bf234521_calibrate_871041_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:37 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:38 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:38 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:39 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:40 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:40 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:41 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f9f3d65a00e3f370267e855564deb8 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash f79dcecea03d7e70dfa8619477d6726d parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/4fd4a8e0-84cd-4b41-b8f9-1c6d81de2c48_calibrate_871043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/4fd4a8e0-84cd-4b41-b8f9-1c6d81de2c48_calibrate_871043_7.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/728b0881-8e77-4ccb-8d2d-c57729801a0a_characterizeImage_871043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/728b0881-8e77-4ccb-8d2d-c57729801a0a_characterizeImage_871043_7.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfba67b23f0a49084f58249ab7606883 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c6c68835-4358-428f-a6f0-ecc2f9d766b3_calibrate_871041_49.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c6c68835-4358-428f-a6f0-ecc2f9d766b3_calibrate_871041_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:41 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e55c3b4f42b70075db5cc511416223ff with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/78feca8d-b1c6-4a25-85df-77c5e4557a2c_calibrate_871043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/78feca8d-b1c6-4a25-85df-77c5e4557a2c_calibrate_871043_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:42 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc3c568700260696da76b905f3332227 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash c4c60331f3ea0c459928593f5732fe0f parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca842d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2516856e-b655-43b7-b866-d3641a504e8c_calibrate_871042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2516856e-b655-43b7-b866-d3641a504e8c_calibrate_871042_14.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7d1140d-9ae2-409c-84fa-0ea10b1e5b63_characterizeImage_871042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/b7d1140d-9ae2-409c-84fa-0ea10b1e5b63_characterizeImage_871042_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:43 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:43 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:44 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:44 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 135 parsl.dataflow.strategy DEBUG: Executor multi has 135 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 135 parsl.dataflow.strategy DEBUG: Executor multi has 135 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5f2e0644892761c2acf846a1b30344c with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/732f6339-2e90-4e23-a8d2-1191c8e03e2a_calibrate_871042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/732f6339-2e90-4e23-a8d2-1191c8e03e2a_calibrate_871042_24.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d3b12ab1a7ed8c09ab0dafa2acb134b with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/beae93a2-62fa-42b4-afdc-bb25ccdb52c7_calibrate_871042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/beae93a2-62fa-42b4-afdc-bb25ccdb52c7_calibrate_871042_8.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee4a95a0efeb5f98d5d066099f0b973 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c7125700-d86a-488e-964d-6f96f25e76b6_calibrate_871042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c7125700-d86a-488e-964d-6f96f25e76b6_calibrate_871042_35.stderr 2024-03-10 15:41:50 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:41:50 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:51 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55ae0ce2fa4df88b2e6109eff63d2d0d with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/f4ef9450-3909-4de5-98bf-06e96c72aec3_calibrate_871041_38.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/f4ef9450-3909-4de5-98bf-06e96c72aec3_calibrate_871041_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:51 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e08ca895ef61e3a243173b074119d84 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/6e27675e-9e99-46ba-ad55-6b0ee9bad1de_calibrate_871042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/6e27675e-9e99-46ba-ad55-6b0ee9bad1de_calibrate_871042_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:53 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48d0559081626a31364af4a9d75c6b04 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c3bbc70e-de57-41fd-8967-3d693d8df533_calibrate_871042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c3bbc70e-de57-41fd-8967-3d693d8df533_calibrate_871042_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:53 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:54 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:54 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:55 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:55 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:56 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:56 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:57 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:57 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84659e05b80f89133ba4f858cb104411 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1e8ddb26-593d-4035-9651-27ad5dd222ab_calibrate_871042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/1e8ddb26-593d-4035-9651-27ad5dd222ab_calibrate_871042_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:58 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:58 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 783ac2d9ef8a970f5b84270820bc49ab with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f640a9c-3cbc-405d-a2db-04f80d087254_calibrate_871042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9f640a9c-3cbc-405d-a2db-04f80d087254_calibrate_871042_27.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c850ef743139b6248ec0e3a8daccf0c0 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 7ff805c8e26fde32c30c167445060668 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b2aff0df-544d-44a8-94e3-40848a1c174b_calibrate_871043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b2aff0df-544d-44a8-94e3-40848a1c174b_calibrate_871043_33.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1962f118-d664-4ab3-a7f3-12b2d08af253_characterizeImage_871043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/1962f118-d664-4ab3-a7f3-12b2d08af253_characterizeImage_871043_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:59 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8a8120677bdc4b2d1a1d670773c51db with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/64af761c-77d1-4f87-b51a-fbd4f8e22da6_calibrate_871042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/64af761c-77d1-4f87-b51a-fbd4f8e22da6_calibrate_871042_59.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e8db0faebb4fffda1760a0ed23d23e7 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/29085662-19a6-4191-82dd-b965fa0ae2ff_calibrate_871043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/29085662-19a6-4191-82dd-b965fa0ae2ff_calibrate_871043_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:41:59 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a623a1fc59dcf3e3bb5508371351fd9a with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/fa951d4f-5905-4344-ad30-bed71242c60e_calibrate_871043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/fa951d4f-5905-4344-ad30-bed71242c60e_calibrate_871043_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:00 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:01 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:01 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:02 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:02 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:03 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1f9c03035eb7b90092d316061f1cee with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3088fb9f-091c-402f-8666-86aa0387b199_calibrate_871043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3088fb9f-091c-402f-8666-86aa0387b199_calibrate_871043_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:04 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5123c3227c4717d8b4b1f145a0e7feb with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/86a7d7a7-08f1-4d09-9cc3-7fb463354e0c_calibrate_871043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/86a7d7a7-08f1-4d09-9cc3-7fb463354e0c_calibrate_871043_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:04 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:05 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:05 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dccab79c5c9f58ace44f5b42604cf45e with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/04e257e4-402d-43c1-a0f5-5d6c37efd91f_calibrate_871042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/04e257e4-402d-43c1-a0f5-5d6c37efd91f_calibrate_871042_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:06 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:06 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:07 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 462257214d3ed97854183a43b19e1bd0 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/32a45141-e338-4e31-bc00-02c697e0d156_calibrate_871042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/32a45141-e338-4e31-bc00-02c697e0d156_calibrate_871042_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:07 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:08 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:08 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:09 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea77d9f353869896c25621f58a739904 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/50a0bb07-5ec9-4c7a-8b06-0be7b97db68f_calibrate_871043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/50a0bb07-5ec9-4c7a-8b06-0be7b97db68f_calibrate_871043_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:09 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:10 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e292f6a6ba40049ec8dc91953df1315e with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/9e72670b-77a6-4a03-b5a0-0a1db62570a2_calibrate_871043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/9e72670b-77a6-4a03-b5a0-0a1db62570a2_calibrate_871043_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:10 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:11 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:12 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:12 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d512cfe7f6423b21980cc3fc8f217970 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9489d51b-4a40-4420-9354-bb4d40f0813f_calibrate_871042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9489d51b-4a40-4420-9354-bb4d40f0813f_calibrate_871042_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:13 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:13 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:14 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fd0656e9a4e885bcfc5c6f4a33f1586 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a7d2159c-8acd-4bf0-8bdd-70bd96ddb706_calibrate_871042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a7d2159c-8acd-4bf0-8bdd-70bd96ddb706_calibrate_871042_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:14 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:15 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:15 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 116 parsl.dataflow.strategy DEBUG: Executor multi has 116 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 116 parsl.dataflow.strategy DEBUG: Executor multi has 116 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91163fd9e003f1df5c7991ef219032f5 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6e87e311-4768-4651-bba7-197cac8d9353_calibrate_871041_59.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6e87e311-4768-4651-bba7-197cac8d9353_calibrate_871041_59.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4ff3cb0f4e4a186d7fc19dd4efe7202 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a5600f0c-f8e3-4380-b9eb-3729aaf20fe6_calibrate_871042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a5600f0c-f8e3-4380-b9eb-3729aaf20fe6_calibrate_871042_7.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f26b7d647e635ba10625f9d7d88abc0c with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/90d67a4c-3c5c-43a3-9cb1-f25bb5110cbc_calibrate_871043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/90d67a4c-3c5c-43a3-9cb1-f25bb5110cbc_calibrate_871043_34.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63ec31a6fffbc3ed356eaeafef7862b4 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/18550b21-27c5-4ba4-8783-f7135a7a7601_calibrate_871043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/18550b21-27c5-4ba4-8783-f7135a7a7601_calibrate_871043_59.stderr 2024-03-10 15:42:21 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:42:21 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:22 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:22 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e31ed8ef647f4efba7fc77dd235c14 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/105b947b-c483-4333-b46c-076ce908045a_calibrate_871042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/105b947b-c483-4333-b46c-076ce908045a_calibrate_871042_56.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d810f05031be36ed894298659ca3acd3 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/18509e18-2187-4a71-a582-cc4e96bf6965_calibrate_871041_35.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/18509e18-2187-4a71-a582-cc4e96bf6965_calibrate_871041_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:23 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43d0d8d61ca29ff21214462cf034fd33 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e94d4e64-04ed-4615-b4e9-02f05918abfb_calibrate_871042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e94d4e64-04ed-4615-b4e9-02f05918abfb_calibrate_871042_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:23 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:24 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:25 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:25 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:26 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:27 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:27 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:28 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:28 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:29 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:30 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d5c6ba40ae94e60e84a9443107d2394 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash f8d631e510062d4360329c18862aa12b parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0951a88-86df-48e9-8028-900157897db3_calibrate_871042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0951a88-86df-48e9-8028-900157897db3_calibrate_871042_34.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/448eaa63-5f06-4b4d-8ad4-17b6c8ca1811_characterizeImage_871042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/448eaa63-5f06-4b4d-8ad4-17b6c8ca1811_characterizeImage_871042_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:30 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:31 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:32 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 481 try 1 failed parsl.dataflow.dflow ERROR: Task 481 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 04dc4e3db6e55a8dd0b5b042888de5a5 with result from task 481 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 482 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 482 try 0 failed parsl.dataflow.dflow INFO: Task 482 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/18984851-581a-4364-b9f3-d704b01ac1f9_calibrate_871042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/18984851-581a-4364-b9f3-d704b01ac1f9_calibrate_871042_2.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/3925b0ca-5fdc-4811-81b0-0780c4298564_characterizeImage_871042_2.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70c239afa1c94a421faf2b934426d9c with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ccd630f-f644-4b10-8f89-0aa1dafad2e1_calibrate_871043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ccd630f-f644-4b10-8f89-0aa1dafad2e1_calibrate_871043_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:32 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a467639dfd43fa106b53c406293fd539 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1845f55-f290-4773-82c2-9d32963a457d_calibrate_871042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1845f55-f290-4773-82c2-9d32963a457d_calibrate_871042_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:33 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:33 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:34 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:34 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94e0000b491b1a3e5464b98535cbe31f with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86852f05-cd6f-45b1-881c-8069576a746a_calibrate_871041_6.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86852f05-cd6f-45b1-881c-8069576a746a_calibrate_871041_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:35 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:36 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f52c10ec6595a3e0fd21aca32da072 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a2592f92-c30b-459d-9edd-04013cd9b321_calibrate_871042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a2592f92-c30b-459d-9edd-04013cd9b321_calibrate_871042_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:36 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a21f2f8ab418d1ab13dd13479ce18471 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/dcf2f6a1-f1f2-413d-9e48-07953f5c6e30_calibrate_871042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/dcf2f6a1-f1f2-413d-9e48-07953f5c6e30_calibrate_871042_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:37 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:37 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:38 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:39 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:39 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 473a3474e8ffce14f5ed4e9f06badcc1 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d9eaa74c-21dc-459d-a816-7665cf57744c_calibrate_871043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d9eaa74c-21dc-459d-a816-7665cf57744c_calibrate_871043_47.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec2a06d19f20d6adceac85e80759a373 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b478d0aa-c6f2-416e-940f-cc94979c0880_calibrate_871043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b478d0aa-c6f2-416e-940f-cc94979c0880_calibrate_871043_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:40 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:40 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:41 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:41 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:42 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:43 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:43 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7a83341ac61e2c45d9245d8696544d with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a6e69d86-198e-4ee4-8f7f-c1f8d0e7c3f0_calibrate_871043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a6e69d86-198e-4ee4-8f7f-c1f8d0e7c3f0_calibrate_871043_42.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8bd81b4978e158b33784a224f9281cc with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7f2d75ab-92e9-4927-835b-eb27ea118bd7_calibrate_871041_42.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7f2d75ab-92e9-4927-835b-eb27ea118bd7_calibrate_871041_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:44 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:44 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:45 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:46 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d61b6de5633c3fe2dda8ca84b69b25f with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1fb78699-b5c5-4664-a099-48bf2d382bad_calibrate_871043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1fb78699-b5c5-4664-a099-48bf2d382bad_calibrate_871043_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:46 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:47 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:47 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:48 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 98 parsl.dataflow.strategy DEBUG: Executor multi has 98 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 98 parsl.dataflow.strategy DEBUG: Executor multi has 98 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62aa34cfb9ffe34eb4926b71440959fa with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ad75767f-0382-451f-8b1e-0f71a4cd2efa_calibrate_871041_16.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ad75767f-0382-451f-8b1e-0f71a4cd2efa_calibrate_871041_16.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc0d81ed168e486206e93b7a8bc5714 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/11638659-f541-4ccc-993b-49cbd3362973_calibrate_871041_28.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/11638659-f541-4ccc-993b-49cbd3362973_calibrate_871041_28.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b41afb15487b660d73bdfea54a557026 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 21148ea64144d9af28575bc498d49e96 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca88ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3ea871a4-7070-4885-ba95-85fb880a3f2f_calibrate_871042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3ea871a4-7070-4885-ba95-85fb880a3f2f_calibrate_871042_39.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5ff696bd-d262-46db-a89c-546c44b6b987_characterizeImage_871042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/5ff696bd-d262-46db-a89c-546c44b6b987_characterizeImage_871042_39.stderr 2024-03-10 15:42:53 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:42:54 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:54 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:55 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c96ecf2245365a2ce7f40fa979615bc2 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94a2bdf6-3791-49dc-84ee-a9ae190c2a11_calibrate_871043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/94a2bdf6-3791-49dc-84ee-a9ae190c2a11_calibrate_871043_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:56 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f91e7091431350451f5527dbde3d51c4 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/96ff37f4-69ee-48f6-a8c3-98f56e3f2bf9_calibrate_871041_22.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/96ff37f4-69ee-48f6-a8c3-98f56e3f2bf9_calibrate_871041_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:56 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:57 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:57 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:58 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:58 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:59 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:42:59 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9179727d6a488969b2c69f75ee37e77e with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/15a20ca3-2972-4807-aaad-6da2eb210d3c_calibrate_871043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/15a20ca3-2972-4807-aaad-6da2eb210d3c_calibrate_871043_31.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3741171a85e86dbf523b63cee7912a03 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/e48b4cf6-333a-4323-8b2a-5148ed423d7b_calibrate_871043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/e48b4cf6-333a-4323-8b2a-5148ed423d7b_calibrate_871043_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:00 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:00 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:01 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:01 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:02 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a17f4f03b7414b09b24e06818461052b with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/187e0215-ae7d-4805-b7f3-2efade41edc0_calibrate_871041_4.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/187e0215-ae7d-4805-b7f3-2efade41edc0_calibrate_871041_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:02 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 856a1283fb73c59d2f16740b6c221c1e with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/aada3442-ccf2-4652-b105-72656fa08dc6_calibrate_871042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/aada3442-ccf2-4652-b105-72656fa08dc6_calibrate_871042_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:03 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:03 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:04 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:04 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:05 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a101756686a24ed6ea76909637391840 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ff94f95-787e-409c-98a9-c449840e9575_calibrate_871043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1ff94f95-787e-409c-98a9-c449840e9575_calibrate_871043_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:05 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:06 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:06 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:07 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:07 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:08 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:08 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:09 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:10 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:10 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:11 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5727bde646810667743199eaee0f7f85 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/0367352f-2141-4d9e-b19e-f5ef1210402e_calibrate_871042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/0367352f-2141-4d9e-b19e-f5ef1210402e_calibrate_871042_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:11 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da207904b94db4b371fd81287515c227 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5f7b10d0-fe92-47ea-a933-03e84b49ecb1_calibrate_871041_34.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/5f7b10d0-fe92-47ea-a933-03e84b49ecb1_calibrate_871041_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:12 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:13 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:13 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abcbd2c3ee7dbe57473de4bcd6408903 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a81fc821-b5d6-4a0a-8009-e0b4d5795382_calibrate_871043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/a81fc821-b5d6-4a0a-8009-e0b4d5795382_calibrate_871043_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:14 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:14 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:15 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:15 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ca9ebd0e1f304cfe316c5dd6c085b76 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/64a04a69-cd27-4f32-8947-0d2b5af75896_calibrate_871041_46.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/64a04a69-cd27-4f32-8947-0d2b5af75896_calibrate_871041_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:16 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:17 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:17 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:18 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dad622b29d6185cad2feb9009639c0ef with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1eac9c0a-ba0f-4b46-b401-af3663136114_calibrate_871043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/1eac9c0a-ba0f-4b46-b401-af3663136114_calibrate_871043_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:18 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 84 parsl.dataflow.strategy DEBUG: Executor multi has 84 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 84 parsl.dataflow.strategy DEBUG: Executor multi has 84 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 196b03c1d469b2ed69da49b9cdfb691c with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/708fd9e5-aae0-4ba4-933f-687ccd6b350c_calibrate_871041_52.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/708fd9e5-aae0-4ba4-933f-687ccd6b350c_calibrate_871041_52.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8c72c1de7c88e893a7ee68418ff42ab with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d2012912-400f-4663-a3dd-219afbbc924f_calibrate_871041_10.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d2012912-400f-4663-a3dd-219afbbc924f_calibrate_871041_10.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adbb695dc7055d8ed04522c58e10e9b9 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b3980c23-28c2-41be-840e-07b56b942ce3_calibrate_871042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b3980c23-28c2-41be-840e-07b56b942ce3_calibrate_871042_52.stderr 2024-03-10 15:43:24 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:43:24 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4b8773f68259f4326cee1a3da3c4737 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/821ee656-ec6e-4034-8ef2-714670fc62ba_calibrate_871043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/821ee656-ec6e-4034-8ef2-714670fc62ba_calibrate_871043_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:25 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71b1fb38f1ff7fa9cb6d05ff720c0c5e with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b9786428-448c-47f6-8bf0-c06d76bf396b_calibrate_871041_25.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b9786428-448c-47f6-8bf0-c06d76bf396b_calibrate_871041_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:25 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de52f64b0b6f82838c00a014e35b2031 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ae8b5c80-b37d-45df-9034-8a43dfcced6f_calibrate_871041_48.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ae8b5c80-b37d-45df-9034-8a43dfcced6f_calibrate_871041_48.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cb6b447cd5fb1ac23fa412d3ab78bdf with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d377c233-963f-4f8f-a200-2b0e3e10a583_calibrate_871041_60.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d377c233-963f-4f8f-a200-2b0e3e10a583_calibrate_871041_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:26 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:26 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:27 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:28 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:28 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:29 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:29 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:30 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:30 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:31 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:32 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cbd4763855f405795f6c65dfb23aff3 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f1244a82-ac97-49e7-8a31-e5260af21607_calibrate_871043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f1244a82-ac97-49e7-8a31-e5260af21607_calibrate_871043_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:32 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:33 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:33 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:34 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2b11c35f296b351878bb80a7a3b0972 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/71fa787f-2683-41f3-af4a-fdebb3f321ce_calibrate_871041_3.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/71fa787f-2683-41f3-af4a-fdebb3f321ce_calibrate_871041_3.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb63444b4992dfc9fffc0c033c0dea80 with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/481f030f-0fa4-473e-b05f-b544022ca06f_calibrate_871041_7.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/481f030f-0fa4-473e-b05f-b544022ca06f_calibrate_871041_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:34 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b3e18edb3c477ec82d275551c29a6e9 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/764f0a9f-2b1f-42f0-b213-346228f4cf2d_calibrate_871042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/764f0a9f-2b1f-42f0-b213-346228f4cf2d_calibrate_871042_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:35 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:36 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 398ffe7b9ccdcbdc7f06a0cfc8e3fe08 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/970fd6b3-2942-49ca-b1cc-0eef4aba4224_calibrate_871042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/970fd6b3-2942-49ca-b1cc-0eef4aba4224_calibrate_871042_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:36 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:37 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:37 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:38 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:38 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8e00b23733c11452d6bf1773a5239ff with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ed648759-b044-418d-bc86-281db82cb457_calibrate_871042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ed648759-b044-418d-bc86-281db82cb457_calibrate_871042_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:39 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:39 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:40 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6bc43d84d2a609f3f471ba2567a9d3d with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/06d9681e-e8bd-42df-8954-06ed15438538_calibrate_871042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/06d9681e-e8bd-42df-8954-06ed15438538_calibrate_871042_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:40 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:41 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:41 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 103c6e6099326708470ad2483035dd5a with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca95ee70-eccf-4eb8-b5f2-2eb506341509_calibrate_871041_12.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca95ee70-eccf-4eb8-b5f2-2eb506341509_calibrate_871041_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:42 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:42 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4035972f91b6c2fbc8ea20d5d869dc5a with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b750007-f860-4c64-9061-f62f6b2817cd_calibrate_871041_8.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b750007-f860-4c64-9061-f62f6b2817cd_calibrate_871041_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:43 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:44 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:44 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:45 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:45 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:46 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:46 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de92a0d4b8b8cb0b443a11783d5d422d with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/34d6f053-8338-461e-82e7-772a0acc5e1c_calibrate_871043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/34d6f053-8338-461e-82e7-772a0acc5e1c_calibrate_871043_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:47 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:48 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e42e208b802fe08cd686136a0b8a7cfc with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f84be570-9324-4871-be41-b545527d12f0_calibrate_871042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/f84be570-9324-4871-be41-b545527d12f0_calibrate_871042_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:48 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:49 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:49 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 66 parsl.dataflow.strategy DEBUG: Executor multi has 66 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 66 parsl.dataflow.strategy DEBUG: Executor multi has 66 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 159a657dd1b9e77a3370fd3228abc012 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/549e0e36-14b9-4126-a12d-a1e74ed62989_calibrate_871042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/549e0e36-14b9-4126-a12d-a1e74ed62989_calibrate_871042_29.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f35e3e1e544cb825cfd5b9948edd5d12 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/804a8201-4c46-4dec-86d2-146b6b3f30bb_calibrate_871043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/804a8201-4c46-4dec-86d2-146b6b3f30bb_calibrate_871043_3.stderr 2024-03-10 15:43:55 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:43:55 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e3a11383abf41de800773ede2f91ca1 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86de14e2-c92a-4e5c-a21b-5a4a56005a85_calibrate_871041_21.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86de14e2-c92a-4e5c-a21b-5a4a56005a85_calibrate_871041_21.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e601814a59aaa9ad5c51dabb25edf852 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/70160937-4461-4605-87d7-cc07ebeb5db4_calibrate_871043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/70160937-4461-4605-87d7-cc07ebeb5db4_calibrate_871043_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:56 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9abd400cfbc854c7e3b8b83bcabbf46 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a5a8067d-1d0f-4184-869f-528e4645e396_calibrate_871041_29.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/a5a8067d-1d0f-4184-869f-528e4645e396_calibrate_871041_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:56 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:57 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80b1f331745747229d69edb5b8d9e944 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f43bacfa-741d-4c9d-931d-cacbd4dda7bb_calibrate_871043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/f43bacfa-741d-4c9d-931d-cacbd4dda7bb_calibrate_871043_44.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b72d4d880f9062fb9d13bacc4778bb36 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6dfa7ac8-733a-4372-8312-d872496d6f22_calibrate_871043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/6dfa7ac8-733a-4372-8312-d872496d6f22_calibrate_871043_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:57 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:58 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:59 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:43:59 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:00 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:00 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:01 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:01 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:02 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1359c374cdb20d0e1a1722f97cf8cf with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b8cc16dc-d453-4661-ad66-2f473cd21d22_calibrate_871043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b8cc16dc-d453-4661-ad66-2f473cd21d22_calibrate_871043_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:03 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:03 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:04 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a7d5590efe6b41894d615c8aad9341e with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b783c40f-39f1-4aff-84dc-97995af08de9_calibrate_871041_47.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b783c40f-39f1-4aff-84dc-97995af08de9_calibrate_871041_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:04 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:05 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bff46e037ae70e3cc3d623028c3827ff with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/16c883bf-e955-41c9-98ba-abf0619d2a2b_calibrate_871043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/16c883bf-e955-41c9-98ba-abf0619d2a2b_calibrate_871043_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:05 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:06 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2780b9469103f5ce8d948230b1d04794 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/3fed0b54-b165-4715-baf3-8f3ef59a7323_calibrate_871041_24.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/3fed0b54-b165-4715-baf3-8f3ef59a7323_calibrate_871041_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:06 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4617d62743f423bf1c5285e77a1860 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/285a39b1-6d72-4077-a144-2adeedd946f2_calibrate_871041_62.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/285a39b1-6d72-4077-a144-2adeedd946f2_calibrate_871041_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:07 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:08 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:08 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:09 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a326a75ba4a9c892b720e66be25d997 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6c004ac6-d923-4e18-b8cb-272cd2a9399e_calibrate_871041_23.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6c004ac6-d923-4e18-b8cb-272cd2a9399e_calibrate_871041_23.stderr parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45406deb8f7f42c37397f627a84a415a with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/45f9e681-49eb-4a10-a848-657723d6f25a_calibrate_871042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/45f9e681-49eb-4a10-a848-657723d6f25a_calibrate_871042_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:09 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:10 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:11 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:11 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a7f107f9fe1fc87ba3afc8d44da6aff with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9018e033-f706-4c78-8f68-1f1d44760d45_calibrate_871042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/9018e033-f706-4c78-8f68-1f1d44760d45_calibrate_871042_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:12 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b12afe045ce354e3c921feeaf53f385 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca92000c-e43c-4ecc-8eb1-1a8b50f621bf_calibrate_871041_20.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/ca92000c-e43c-4ecc-8eb1-1a8b50f621bf_calibrate_871041_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:12 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:13 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9a785dfc044b463482f117da9bb62c9 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86ab82e0-2a53-47ff-b267-259bbf8011fb_calibrate_871041_37.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/86ab82e0-2a53-47ff-b267-259bbf8011fb_calibrate_871041_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:14 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:14 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5029d5db7b708267f0a92ba477a37be with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/23886196-662d-491d-9950-76f1fb94d94b_calibrate_871041_45.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/23886196-662d-491d-9950-76f1fb94d94b_calibrate_871041_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:15 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d101a83b28744662795b2729cb107eff with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/25b89215-758d-4b58-95a3-fa991e6cb2dd_calibrate_871041_50.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/25b89215-758d-4b58-95a3-fa991e6cb2dd_calibrate_871041_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:16 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:16 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:17 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:17 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:18 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:19 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:19 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:20 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4287bde667eaa509d8e891aa063ac232 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5eedec86-a83a-40b4-ab78-466c07b2942f_calibrate_871043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/5eedec86-a83a-40b4-ab78-466c07b2942f_calibrate_871043_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:20 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:21 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:21 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec60876d84b57f5923e639c74a42ff6 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0c992e25-7927-4aba-938d-e3f7276e8877_calibrate_871043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0c992e25-7927-4aba-938d-e3f7276e8877_calibrate_871043_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 45 parsl.dataflow.strategy DEBUG: Executor multi has 45 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 45 parsl.dataflow.strategy DEBUG: Executor multi has 45 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0addd09c259ac3d7ec481a18017cc445 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7480f460-a43f-479b-ba1e-7b1f77722450_calibrate_871041_11.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/7480f460-a43f-479b-ba1e-7b1f77722450_calibrate_871041_11.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3ceca32639511b1aa0f69cd89e86e0a with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e8d348e0-48e1-41a9-bb80-79797105bd0d_calibrate_871041_1.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/e8d348e0-48e1-41a9-bb80-79797105bd0d_calibrate_871041_1.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67a8cf9495585d56e1345edcd09c7ba0 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/46f47ab6-3e61-4261-97ab-9805a7c193f2_calibrate_871042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/46f47ab6-3e61-4261-97ab-9805a7c193f2_calibrate_871042_19.stderr 2024-03-10 15:44:27 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:44:27 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:27 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:27 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:28 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa12cd501c7f521004e2876985c429ff with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d35c6298-eadf-467c-ba68-61d386de726c_calibrate_871043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d35c6298-eadf-467c-ba68-61d386de726c_calibrate_871043_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:28 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:28 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:29 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:29 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:29 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:29 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a26e5bec23a38d769ae9c6b234d219c with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/09e6ef83-7b84-4615-95d2-2942d85dbced_calibrate_871041_58.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/09e6ef83-7b84-4615-95d2-2942d85dbced_calibrate_871041_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:30 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:30 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:30 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:31 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:31 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:31 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:32 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4593ecc7614599f5710bb5097d51216d with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8a327a39-1ce4-4436-a89e-796769832460_calibrate_871041_39.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/8a327a39-1ce4-4436-a89e-796769832460_calibrate_871041_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:32 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:32 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:33 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:33 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:33 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:34 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29e636d5ccccde35dbcb2bdb6e78e3c5 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e2f0ec17-25be-4c68-b357-f73b3d34d20d_calibrate_871042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/e2f0ec17-25be-4c68-b357-f73b3d34d20d_calibrate_871042_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:34 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:34 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:35 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:35 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:35 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:35 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:36 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:36 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:36 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:37 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dc27b177e05ef0fcd0ebd9fc1a19165 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/84094d57-b4e5-45d3-8e25-b231ac2e2596_calibrate_871043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/84094d57-b4e5-45d3-8e25-b231ac2e2596_calibrate_871043_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:37 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:37 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:37 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:38 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:38 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:38 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:39 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:39 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a7ec758bd2bdf277eb7a91fc7d70274 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/815b9560-b659-49cc-b55e-036d9cc559fb_calibrate_871043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/815b9560-b659-49cc-b55e-036d9cc559fb_calibrate_871043_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:39 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:40 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:40 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f569ee2b9c6dd888e4e1cba269e61ae1 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b7143b31-f45d-48a4-9401-d18a0f197da0_calibrate_871041_17.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/b7143b31-f45d-48a4-9401-d18a0f197da0_calibrate_871041_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:40 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:44:40 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 35 parsl.dataflow.strategy DEBUG: Executor multi has 35 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 35 parsl.dataflow.strategy DEBUG: Executor multi has 35 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f774e9fc16c7fa103e5a7648602d3870 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash a5f7d558fc096f31968c573c9cb0eb07 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb97f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/c45cc253-1461-4be9-8bab-68985f39a9ae_calibrate_871043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/c45cc253-1461-4be9-8bab-68985f39a9ae_calibrate_871043_61.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b636902f-38bb-47f5-8cba-20665b05a129_characterizeImage_871043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871043/b636902f-38bb-47f5-8cba-20665b05a129_characterizeImage_871043_61.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 518e7978134620b63a755b45d371c935 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fbabdbb8-faaa-48e2-89d8-7771910e38e4_calibrate_871042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/fbabdbb8-faaa-48e2-89d8-7771910e38e4_calibrate_871042_10.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 34 parsl.dataflow.strategy DEBUG: Executor multi has 34 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dbe8df763fd64cfa977a096c439315a with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/afcc7f0c-d44e-446e-9021-a378f95c6513_calibrate_871041_40.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/afcc7f0c-d44e-446e-9021-a378f95c6513_calibrate_871041_40.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 33 parsl.dataflow.strategy DEBUG: Executor multi has 33 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a032eda2c16c94cff3dabd65584a2f6 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ea0786a9-09d3-49e9-b7e0-74928ec14a49_calibrate_871042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/ea0786a9-09d3-49e9-b7e0-74928ec14a49_calibrate_871042_41.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 562495ee244660ccf4ad17aa465ed16f with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a4821404-9db9-4560-b34b-b8f872aa5efc_calibrate_871042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a4821404-9db9-4560-b34b-b8f872aa5efc_calibrate_871042_18.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d071de99f0d969996dcdba9e5ec227 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1f04832-9be7-4e02-ade1-1a5f0fbb465c_calibrate_871042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b1f04832-9be7-4e02-ade1-1a5f0fbb465c_calibrate_871042_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 30 parsl.dataflow.strategy DEBUG: Executor multi has 30 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4395cd8bb081c33254ffd810eb6990cd with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/076557d5-37bd-4912-8454-44b9e833a7a9_calibrate_871043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/076557d5-37bd-4912-8454-44b9e833a7a9_calibrate_871043_35.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8a800e59254ba3ae276b0f9ef02d652 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/11e43bcb-57ac-4f09-a723-4c27a7a76a57_calibrate_871043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/11e43bcb-57ac-4f09-a723-4c27a7a76a57_calibrate_871043_8.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9ae35e7134c644768db12a20c394576 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6ab4ed3b-6743-4694-8733-6936beaf7ef4_calibrate_871041_9.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/6ab4ed3b-6743-4694-8733-6936beaf7ef4_calibrate_871041_9.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 965876de713349b323c9f85c66abe688 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d555e65a-d460-43db-8bd0-320eb497c54c_calibrate_871041_51.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/d555e65a-d460-43db-8bd0-320eb497c54c_calibrate_871041_51.stderr 2024-03-10 15:45:01 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e11e6bc22592734d92aa6be0bf0485e with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b188633-eb0b-4f61-914d-e69808817f6d_calibrate_871041_15.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/1b188633-eb0b-4f61-914d-e69808817f6d_calibrate_871041_15.stderr 2024-03-10 15:45:01 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:01 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:01 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:01 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:02 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:02 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d92ab6cd69bccac82a619d9947aa8609 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c610941a-9283-4dc5-adbc-2f5fd5f1ab01_calibrate_871041_13.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/c610941a-9283-4dc5-adbc-2f5fd5f1ab01_calibrate_871041_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:02 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:03 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:03 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:03 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:03 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:04 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:04 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:04 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:04 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:05 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:05 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:05 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:05 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:06 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:06 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:06 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:06 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:07 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5f2a2c406335cfa26289e86626c8a1d with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c846677a-3f74-46f3-8b35-5133b515005c_calibrate_871042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/c846677a-3f74-46f3-8b35-5133b515005c_calibrate_871042_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:07 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70c8bd816973b36ddcb9316d4ac43f7f with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a59368f8-1733-4298-85a7-dcb898f76279_calibrate_871042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/a59368f8-1733-4298-85a7-dcb898f76279_calibrate_871042_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:07 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:08 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:08 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:08 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:08 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:09 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:09 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:09 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:09 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8095edb1ac6dcb386ece03e2ab3098fa with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3d969d05-5234-477e-959f-ec53e7b3eb06_calibrate_871043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/3d969d05-5234-477e-959f-ec53e7b3eb06_calibrate_871043_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:10 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d06f8dc641b3d44f7b844e7af4bc48c4 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/9e9719aa-1497-4cad-b511-e678c32c7340_calibrate_871041_14.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/9e9719aa-1497-4cad-b511-e678c32c7340_calibrate_871041_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:10 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:10 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:10 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:11 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:11 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:11 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:11 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:12 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:12 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:12 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9307779c7d5c2ca7862639d6166ef1e2 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/41e24fa1-3c9a-4afe-8ebb-6f952b0e0354_calibrate_871041_41.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/41e24fa1-3c9a-4afe-8ebb-6f952b0e0354_calibrate_871041_41.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0e5c7a0ea0142b2b3adebba74dcebda with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/cafe4761-0c96-4843-b346-cf0a5842b114_calibrate_871043_24.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/cafe4761-0c96-4843-b346-cf0a5842b114_calibrate_871043_24.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:12 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 18 parsl.dataflow.strategy DEBUG: Executor multi has 18 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 18 parsl.dataflow.strategy DEBUG: Executor multi has 18 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3b6413484c1c753aa5601fbde5dd4ef with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/8d867a21-3b26-4e37-8592-316657932738_calibrate_871043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/8d867a21-3b26-4e37-8592-316657932738_calibrate_871043_55.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5d8fc1c665609e437a4662c5f3307f2 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/cf7b59e5-7eef-4b52-b7e7-0da9e7dad0ed_calibrate_871041_27.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/cf7b59e5-7eef-4b52-b7e7-0da9e7dad0ed_calibrate_871041_27.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f79dcecea03d7e70dfa8619477d6726d with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/4fd4a8e0-84cd-4b41-b8f9-1c6d81de2c48_calibrate_871043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/4fd4a8e0-84cd-4b41-b8f9-1c6d81de2c48_calibrate_871043_7.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21148ea64144d9af28575bc498d49e96 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3ea871a4-7070-4885-ba95-85fb880a3f2f_calibrate_871042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/3ea871a4-7070-4885-ba95-85fb880a3f2f_calibrate_871042_39.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 14 parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 394c8e686c914c4335d93514094c08e5 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0f4836ca-971c-437f-94de-1ccd8a833d98_calibrate_871043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0f4836ca-971c-437f-94de-1ccd8a833d98_calibrate_871043_1.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ff805c8e26fde32c30c167445060668 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b2aff0df-544d-44a8-94e3-40848a1c174b_calibrate_871043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/b2aff0df-544d-44a8-94e3-40848a1c174b_calibrate_871043_33.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4c60331f3ea0c459928593f5732fe0f with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2516856e-b655-43b7-b866-d3641a504e8c_calibrate_871042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/2516856e-b655-43b7-b866-d3641a504e8c_calibrate_871042_14.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70f25f85ee0b2a8d0e2538eb6c2944aa with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0fd99b48-a4cf-4b44-920a-598c399b47cd_calibrate_871043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0fd99b48-a4cf-4b44-920a-598c399b47cd_calibrate_871043_32.stderr 2024-03-10 15:45:33 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:45:33 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:33 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:33 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:34 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:34 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:34 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:34 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:35 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:35 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80efc366981764de06c2d5c4d6dac5d with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/23942b39-d3f8-47ba-95ea-082264867b2f_calibrate_871043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/23942b39-d3f8-47ba-95ea-082264867b2f_calibrate_871043_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:35 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:35 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:36 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:36 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:36 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:36 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:37 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:37 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:37 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:37 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:38 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:38 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:38 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:38 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:39 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd3aaa93fea01f0914914d853252bc05 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 357a18a7ad14fa1a8cb61cc4c86e254a parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb925080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/813a2c7e-3388-4420-907e-5de79696607f_calibrate_871042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/813a2c7e-3388-4420-907e-5de79696607f_calibrate_871042_61.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd67dac5-316c-47b7-aa2e-a7e131b18850_characterizeImage_871042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/cd67dac5-316c-47b7-aa2e-a7e131b18850_characterizeImage_871042_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:39 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:39 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:39 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:40 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:40 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bde20254238507a562f5f7b9955b5a14 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d8586ef9-1140-4091-b2a3-369395283cb6_calibrate_871043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/d8586ef9-1140-4091-b2a3-369395283cb6_calibrate_871043_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:40 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:40 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:41 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:41 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:41 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:41 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:42 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:42 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:42 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:42 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:43 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:43 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:43 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:43 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:43 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:44 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:45:44 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dc5a44fa98292af08e7251a7ce3b964 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0d3879af-d879-4ff1-8262-540e193639a2_calibrate_871043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/0d3879af-d879-4ff1-8262-540e193639a2_calibrate_871043_20.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae4c9b40a3128f8d4a585845d34a6d7f with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/12d8a490-25ca-4545-a09a-a80210dc14bc_calibrate_871043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/12d8a490-25ca-4545-a09a-a80210dc14bc_calibrate_871043_36.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8d631e510062d4360329c18862aa12b with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0951a88-86df-48e9-8028-900157897db3_calibrate_871042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b0951a88-86df-48e9-8028-900157897db3_calibrate_871042_34.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5f7d558fc096f31968c573c9cb0eb07 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/c45cc253-1461-4be9-8bab-68985f39a9ae_calibrate_871043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871043/c45cc253-1461-4be9-8bab-68985f39a9ae_calibrate_871043_61.stderr 2024-03-10 15:46:04 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:46:04 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:05 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:05 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:05 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:05 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:06 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:06 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:06 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 869f3d56e35ad431add425270ab61bef with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b678b4a9-ec0b-4aee-a3f9-e194a69a0095_calibrate_871042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/b678b4a9-ec0b-4aee-a3f9-e194a69a0095_calibrate_871042_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:06 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:06 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:07 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:07 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:07 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:07 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:08 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:08 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:08 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:08 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:08 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:09 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:09 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:09 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:09 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:09 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:10 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:10 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:10 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:10 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:10 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:11 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:11 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:11 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:11 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:11 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:12 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:12 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:12 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:12 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:12 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:13 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:13 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:13 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:13 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:13 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:14 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:14 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:46:39 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:46:39 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:39 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:40 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:40 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:40 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:40 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:40 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:41 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:41 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:41 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:41 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:41 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:42 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:42 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:42 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:42 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:42 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:43 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:43 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:43 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:43 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:44 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:44 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:44 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:44 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:44 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:45 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:45 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:45 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:45 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:45 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:46 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:46 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:46 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:46 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:46 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:47 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:47 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:47 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:47 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:47 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:48 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:48 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:48 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:48 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:46:49 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 357a18a7ad14fa1a8cb61cc4c86e254a with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/813a2c7e-3388-4420-907e-5de79696607f_calibrate_871042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/813a2c7e-3388-4420-907e-5de79696607f_calibrate_871042_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:47:14 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:47:14 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:14 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:14 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:14 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:15 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:15 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:15 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:15 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:15 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:16 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:16 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:16 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:16 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:17 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:17 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:17 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:17 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:17 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:18 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:18 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:18 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:18 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:19 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:19 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:19 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:19 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:19 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:20 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:20 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:20 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:20 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:20 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:21 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:21 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:21 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:21 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:21 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:22 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:22 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:22 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:22 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:22 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:23 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:23 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:23 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:23 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:47:49 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:47:49 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:49 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:49 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:49 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:50 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:50 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:50 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:50 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:50 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:51 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:51 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:51 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:51 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:51 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:52 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:52 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:52 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:52 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:53 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:53 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:53 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:53 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:53 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:54 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:54 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:54 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:54 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:54 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:55 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:55 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:55 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:55 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:55 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:56 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:56 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:56 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:56 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:56 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:57 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:57 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:57 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:57 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:57 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:58 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:58 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:47:58 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:48:23 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:48:23 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:23 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:24 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:24 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:24 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:24 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:25 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:25 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:25 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:25 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:25 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:26 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:26 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:26 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:26 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:26 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:27 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:27 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:27 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:27 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:27 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:28 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:28 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:28 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:28 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:28 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:29 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:29 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:29 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:29 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:29 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:30 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:30 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:30 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:30 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:31 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:31 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:31 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:31 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:31 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:32 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:32 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:32 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:32 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:32 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:33 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:48:58 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:48:58 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:58 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:58 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:59 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:59 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:59 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:59 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:48:59 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:00 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:00 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:00 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:00 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:00 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:01 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:01 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:01 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:01 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:01 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:02 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:02 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:02 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:02 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:02 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:03 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:03 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:03 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:03 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:03 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:04 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:04 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:04 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:04 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:04 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:05 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:05 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:05 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:05 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:05 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:06 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:06 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:06 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:06 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:06 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:07 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:07 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:07 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:49:32 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:49:32 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:33 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:33 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:33 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:33 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:33 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:34 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:34 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:34 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:34 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:34 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:35 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:35 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:35 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:35 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:35 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:36 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:36 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:36 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:36 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:36 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:37 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:37 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:37 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:37 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:37 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:38 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:38 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:38 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:38 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:39 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:39 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:39 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:39 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:39 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:40 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:40 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:40 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:40 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:40 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:41 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:41 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:41 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:41 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:41 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:49:42 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cb87a939975967c4fb65b76139b92e7 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 72742ee540fa5ef62f53e2fa99febdbb parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1cb901800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/42828087-8111-4fb1-8be2-f43886a5e06f_calibrate_871042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/42828087-8111-4fb1-8be2-f43886a5e06f_calibrate_871042_13.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/4a3eeaf2-3156-4363-9841-e598f8679ada_characterizeImage_871042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871042/4a3eeaf2-3156-4363-9841-e598f8679ada_characterizeImage_871042_13.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:50:07 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:50:07 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:07 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:07 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:07 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:08 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:08 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:08 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:08 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:09 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:09 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:09 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:09 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:09 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:10 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:10 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:10 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:10 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:10 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:11 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:11 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:11 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:11 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:11 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:12 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:12 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:12 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:12 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:12 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:13 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:13 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:13 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:13 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:13 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:14 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:14 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:14 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:14 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:14 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:15 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:15 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:15 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:15 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:15 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:16 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:16 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:16 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:50:41 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:50:41 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:42 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:42 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:42 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:42 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:42 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:43 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:43 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:43 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:43 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:43 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:44 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:44 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:44 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:44 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:44 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:45 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:45 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:45 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:45 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:45 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:46 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:46 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:46 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:46 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:46 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:47 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:47 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:47 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:47 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:48 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:48 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:48 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:48 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:50 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:50 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:50 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:51 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:51 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:51 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:51 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:52 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:52 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72742ee540fa5ef62f53e2fa99febdbb with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/42828087-8111-4fb1-8be2-f43886a5e06f_calibrate_871042_13.stdout parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871042/42828087-8111-4fb1-8be2-f43886a5e06f_calibrate_871042_13.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:52 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:52 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:50:52 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:51:13 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:51:13 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:13 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:13 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:13 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:13 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:14 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:14 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:14 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:14 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:14 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:15 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:15 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:15 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:15 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:15 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:16 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:16 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:16 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:16 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:17 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:17 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:17 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:17 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:17 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:18 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:18 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:18 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:18 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:18 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:19 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:19 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:19 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:19 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:19 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:20 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:20 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:20 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:22 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:22 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:22 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:22 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:23 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:23 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:23 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:23 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:23 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:51:44 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:51:44 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:44 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:44 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:44 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:45 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:45 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:45 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:45 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:45 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:46 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:46 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:46 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:46 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:46 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:47 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:47 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:47 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:47 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:47 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:48 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:48 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:48 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:48 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:49 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:49 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:49 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:49 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:49 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:50 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:50 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:50 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:50 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:50 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:51 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:51 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:51 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:51 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:51 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:52 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:52 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:54 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:54 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:54 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:54 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:55 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:51:55 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:52:15 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:52:15 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:15 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:16 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:16 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:16 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:16 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:16 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:17 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:17 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:17 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:17 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:17 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:18 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:18 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:18 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:18 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:19 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:19 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:19 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:19 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:19 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:20 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:20 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:20 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:20 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:21 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:21 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:21 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:21 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:21 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:22 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:22 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:22 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:22 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:22 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:23 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:23 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:23 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:23 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:23 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:24 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:24 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:24 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:24 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:26 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:27 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:52:47 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:52:47 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:47 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:47 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:47 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:48 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:48 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:48 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:48 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:49 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:49 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:49 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:49 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:49 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:50 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:50 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:50 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:50 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:50 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:51 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:51 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:51 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:51 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:51 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:52 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:52 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:52 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:52 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:52 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:53 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:53 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:53 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:53 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:53 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:54 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:54 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:54 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:54 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:54 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:55 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:55 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:55 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:55 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:55 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:56 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:56 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:52:56 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:53:21 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:53:21 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:22 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:22 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:22 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:22 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:22 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:23 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:23 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:23 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:23 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:23 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:24 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:24 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:24 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:24 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:24 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:25 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:25 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:25 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:25 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:25 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:26 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:26 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:26 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:26 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:27 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:27 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:27 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:27 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:27 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:28 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:28 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:28 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:28 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:28 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:29 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:29 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:29 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:29 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:29 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:30 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:30 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:32 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:32 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:32 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:33 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:53:53 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:53:53 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:53 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:53 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:54 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:54 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:54 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:54 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:54 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:55 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:55 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:55 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:55 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:55 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:56 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:56 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:56 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:56 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:56 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:57 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:57 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:57 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:57 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:58 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:58 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:58 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:58 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:58 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:59 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:59 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:59 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:59 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:53:59 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:00 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:00 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:00 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:00 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:00 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:01 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:01 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:01 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:01 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:01 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:02 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:02 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:02 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:02 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:54:27 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:54:27 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:28 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:28 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:28 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:28 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:29 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:29 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:29 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:29 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:29 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:30 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:30 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:30 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:30 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:31 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:31 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:31 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:31 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:31 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:32 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:32 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:32 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:32 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:32 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:33 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:33 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:33 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:33 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:33 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:34 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:34 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:34 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:34 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:34 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:35 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:35 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:35 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:35 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:35 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:36 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:38 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:38 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:38 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:39 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:39 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:39 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:54:59 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:54:59 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:54:59 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:00 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:00 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:00 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:00 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:01 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:01 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:01 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:01 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:01 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:02 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:02 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:02 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:02 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:02 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:03 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:03 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:03 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:03 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:03 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:04 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:04 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:04 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:04 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:04 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:05 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:05 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:05 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:05 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:06 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:06 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:06 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:06 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:06 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:07 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:07 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:07 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:07 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:07 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:08 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:08 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:08 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:08 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:08 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:09 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:55:31 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:55:31 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:31 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:31 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:31 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:32 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:32 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:32 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:32 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:32 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:33 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:33 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:33 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:33 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:33 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:34 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:34 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:34 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:34 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:34 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:35 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:35 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:35 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:35 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:35 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:36 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:36 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:36 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:36 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:36 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:37 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:37 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:37 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:37 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:37 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:38 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:38 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:38 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:38 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:39 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:39 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:39 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:39 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:39 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:40 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:40 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:55:40 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:56:05 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:56:05 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:05 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:06 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:06 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:06 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:06 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:07 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:07 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:07 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:07 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:07 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:08 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:08 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:08 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:08 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:08 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:09 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:09 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:09 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:09 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:09 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:10 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:10 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:10 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:10 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:10 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:11 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:11 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:11 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:11 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:11 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:12 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:12 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:12 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:12 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:12 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:13 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:13 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:13 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:13 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:13 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:14 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:14 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:14 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:14 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:14 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:56:37 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:56:37 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:37 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:37 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:37 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:37 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:38 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:38 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:38 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:38 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:38 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:39 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:39 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:39 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:39 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:39 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:40 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:40 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:40 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:40 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:41 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:41 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:41 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:41 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:41 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:42 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:42 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:42 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:42 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:42 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:43 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:43 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:43 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:43 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:43 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:44 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:44 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:44 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:44 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:44 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:45 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:45 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:45 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:45 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:45 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:46 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:56:46 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0b95fdef4cc438552733a2e99c25017 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 129919530c4f2dd548c5450fecdf385a parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f1ca7f8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/07877f09-2772-4c08-a737-a6fa0ca246c4_calibrate_871041_61.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/07877f09-2772-4c08-a737-a6fa0ca246c4_calibrate_871041_61.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a4865518-fd7b-4eda-a282-1f8786ede90f_characterizeImage_871041_61.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/characterizeImage/871041/a4865518-fd7b-4eda-a282-1f8786ede90f_characterizeImage_871041_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 2024-03-10 15:57:11 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:57:11 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:11 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:12 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:12 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:12 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:12 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:12 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:13 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:13 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:13 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:13 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:13 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:14 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:14 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:14 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:14 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:14 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:15 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:15 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:15 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:15 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:15 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:16 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:16 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:16 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:16 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:16 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:17 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:17 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:17 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980029,16980030,16980031,16980032,16980033,16980034,16980035,16980036,16980037,16980038,16980039,16980040,16980041,16980042,16980043,16980044,16980045,16980046,16980047,16980048,16980049,16980050,16980051,16980052,16980053,16980054,16980055,16980056,16980057,16980058' parsl.providers.slurm.slurm DEBUG: squeue returned 16980058 R 16980048 R 16980049 R 16980050 R 16980051 R 16980052 R 16980053 R 16980054 R 16980055 R 16980056 R 16980057 R 16980043 R 16980044 R 16980045 R 16980046 R 16980047 R 16980037 R 16980038 R 16980039 R 16980040 R 16980041 R 16980042 R 16980030 R 16980031 R 16980032 R 16980033 R 16980034 R 16980035 R 16980036 R 16980029 R parsl.providers.slurm.slurm DEBUG: Updating job 16980058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980057 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980032 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980029 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:17 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:17 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:18 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:18 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:18 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980059,16980060,16980061,16980062,16980069' parsl.providers.slurm.slurm DEBUG: squeue returned 16980062 R 16980069 R 16980059 R 16980060 R 16980061 R parsl.providers.slurm.slurm DEBUG: Updating job 16980062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980061 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:18 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:19 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:19 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:19 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:19 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:19 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:20 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:20 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:20 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:20 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING 2024-03-10 15:57:20 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16980071,16980072,16980073,16980074,16980075,16980076,16980077,16980078,16980079,16980080,16980081' parsl.providers.slurm.slurm DEBUG: squeue returned 16980081 R 16980074 R 16980075 R 16980076 R 16980077 R 16980078 R 16980079 R 16980080 R 16980071 R 16980072 R 16980073 R parsl.providers.slurm.slurm DEBUG: Updating job 16980081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16980073 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23028791876560 parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 129919530c4f2dd548c5450fecdf385a with result from task 395 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 552 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.failed: 3 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 3 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/07877f09-2772-4c08-a737-a6fa0ca246c4_calibrate_871041_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/submit/DEEP/20190706/B0b/science#step1/20240310T152300Z/logs/calibrate/871041/07877f09-2772-4c08-a737-a6fa0ca246c4_calibrate_871041_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=47, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 47 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2726a154ba9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2726a154ba9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d69eba6e9e73 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d69eba6e9e73 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79c7eecfabd6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79c7eecfabd6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 777246b40a1f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 777246b40a1f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eb04d73308f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eb04d73308f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e49e6bd64a6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e49e6bd64a6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b9df5edb2e34 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b9df5edb2e34 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 144330c3cdfc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 144330c3cdfc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0816e7b39d2d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0816e7b39d2d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 160da0613a08 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 160da0613a08 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 583e6b50ebf1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 583e6b50ebf1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d91a4c5d7d35 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d91a4c5d7d35 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60561a1d1754 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60561a1d1754 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 615c8d48b03a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 615c8d48b03a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 812be4addeb6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 812be4addeb6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 10a424034b4f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 10a424034b4f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5eeb8130618d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5eeb8130618d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d8601329989c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d8601329989c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c91e3117fec5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c91e3117fec5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa3d44a98589 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa3d44a98589 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60395a103a04 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60395a103a04 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 495cea3086ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 495cea3086ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ba3b9e8c7ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ba3b9e8c7ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 21baba7e7a39 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 21baba7e7a39 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eaf73fa053c7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eaf73fa053c7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0def9faaf040 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0def9faaf040 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e5d72298bb0e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e5d72298bb0e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 89bdbce3fc8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 89bdbce3fc8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9ca5ca406e33 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9ca5ca406e33 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8bb01e45281 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8bb01e45281 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 043d34a8e0a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 043d34a8e0a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 898e8e4db068 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 898e8e4db068 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1dbd37fc2557 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1dbd37fc2557 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87838aaa45c8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87838aaa45c8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 06a47b84c39f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 06a47b84c39f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3767d41eeea9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3767d41eeea9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fed0cb87f9b6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fed0cb87f9b6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7826a3ca9fed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7826a3ca9fed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 70dfe5f218f0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 70dfe5f218f0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d4ecb1bab1e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d4ecb1bab1e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd7be58724e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd7be58724e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60c633d4eb1e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60c633d4eb1e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3670c9119ba6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3670c9119ba6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7aede1791235 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7aede1791235 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a42b28ad423f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a42b28ad423f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0e68454af21f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0e68454af21f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9fed78049f54 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9fed78049f54 2024-03-10 15:57:43 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 2024-03-10 15:57:43 proc_lsst.multi:201 [INFO] cancelling 16980075 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980075 on provider escience_a40 2024-03-10 15:57:44 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro 2024-03-10 15:57:44 proc_lsst.multi:201 [INFO] cancelling 16980055 on provider astro proc_lsst.multi INFO: cancelling 16980055 on provider astro 2024-03-10 15:57:44 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro 2024-03-10 15:57:44 proc_lsst.multi:201 [INFO] cancelling 16980057 on provider astro proc_lsst.multi INFO: cancelling 16980057 on provider astro 2024-03-10 15:57:44 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro 2024-03-10 15:57:44 proc_lsst.multi:201 [INFO] cancelling 16980041 on provider astro proc_lsst.multi INFO: cancelling 16980041 on provider astro 2024-03-10 15:57:44 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro 2024-03-10 15:57:44 proc_lsst.multi:201 [INFO] cancelling 16980030 on provider astro proc_lsst.multi INFO: cancelling 16980030 on provider astro 2024-03-10 15:57:44 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro 2024-03-10 15:57:44 proc_lsst.multi:201 [INFO] cancelling 16980038 on provider astro proc_lsst.multi INFO: cancelling 16980038 on provider astro 2024-03-10 15:57:44 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro 2024-03-10 15:57:44 proc_lsst.multi:201 [INFO] cancelling 16980032 on provider astro proc_lsst.multi INFO: cancelling 16980032 on provider astro 2024-03-10 15:57:45 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro 2024-03-10 15:57:45 proc_lsst.multi:201 [INFO] cancelling 16980033 on provider astro proc_lsst.multi INFO: cancelling 16980033 on provider astro 2024-03-10 15:57:45 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro 2024-03-10 15:57:45 proc_lsst.multi:201 [INFO] cancelling 16980035 on provider astro proc_lsst.multi INFO: cancelling 16980035 on provider astro 2024-03-10 15:57:45 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro 2024-03-10 15:57:45 proc_lsst.multi:201 [INFO] cancelling 16980039 on provider astro proc_lsst.multi INFO: cancelling 16980039 on provider astro 2024-03-10 15:57:45 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 2024-03-10 15:57:45 proc_lsst.multi:201 [INFO] cancelling 16980081 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980081 on provider escience_a40 2024-03-10 15:57:45 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro 2024-03-10 15:57:45 proc_lsst.multi:201 [INFO] cancelling 16980031 on provider astro proc_lsst.multi INFO: cancelling 16980031 on provider astro 2024-03-10 15:57:46 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro 2024-03-10 15:57:46 proc_lsst.multi:201 [INFO] cancelling 16980047 on provider astro proc_lsst.multi INFO: cancelling 16980047 on provider astro 2024-03-10 15:57:46 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro 2024-03-10 15:57:46 proc_lsst.multi:201 [INFO] cancelling 16980048 on provider astro proc_lsst.multi INFO: cancelling 16980048 on provider astro 2024-03-10 15:57:46 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro 2024-03-10 15:57:46 proc_lsst.multi:201 [INFO] cancelling 16980049 on provider astro proc_lsst.multi INFO: cancelling 16980049 on provider astro 2024-03-10 15:57:46 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro 2024-03-10 15:57:46 proc_lsst.multi:201 [INFO] cancelling 16980058 on provider astro proc_lsst.multi INFO: cancelling 16980058 on provider astro 2024-03-10 15:57:46 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 2024-03-10 15:57:46 proc_lsst.multi:201 [INFO] cancelling 16980079 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980079 on provider escience_a40 2024-03-10 15:57:47 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro 2024-03-10 15:57:47 proc_lsst.multi:201 [INFO] cancelling 16980052 on provider astro proc_lsst.multi INFO: cancelling 16980052 on provider astro 2024-03-10 15:57:47 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro 2024-03-10 15:57:47 proc_lsst.multi:201 [INFO] cancelling 16980040 on provider astro proc_lsst.multi INFO: cancelling 16980040 on provider astro 2024-03-10 15:57:47 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:57:47 proc_lsst.multi:201 [INFO] cancelling 90235 on provider local proc_lsst.multi INFO: cancelling 90235 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 90235 2024-03-10 15:57:47 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro 2024-03-10 15:57:47 proc_lsst.multi:201 [INFO] cancelling 16980042 on provider astro proc_lsst.multi INFO: cancelling 16980042 on provider astro 2024-03-10 15:57:47 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro 2024-03-10 15:57:47 proc_lsst.multi:201 [INFO] cancelling 16980036 on provider astro proc_lsst.multi INFO: cancelling 16980036 on provider astro 2024-03-10 15:57:47 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 2024-03-10 15:57:47 proc_lsst.multi:201 [INFO] cancelling 16980076 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980076 on provider escience_a40 2024-03-10 15:57:48 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro 2024-03-10 15:57:48 proc_lsst.multi:201 [INFO] cancelling 16980043 on provider astro proc_lsst.multi INFO: cancelling 16980043 on provider astro 2024-03-10 15:57:48 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 2024-03-10 15:57:48 proc_lsst.multi:201 [INFO] cancelling 16980077 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980077 on provider escience_a40 2024-03-10 15:57:48 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k 2024-03-10 15:57:48 proc_lsst.multi:201 [INFO] cancelling 16980061 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980061 on provider escience_rtx6k 2024-03-10 15:57:48 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro 2024-03-10 15:57:48 proc_lsst.multi:201 [INFO] cancelling 16980056 on provider astro proc_lsst.multi INFO: cancelling 16980056 on provider astro 2024-03-10 15:57:48 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 2024-03-10 15:57:48 proc_lsst.multi:201 [INFO] cancelling 16980080 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980080 on provider escience_a40 2024-03-10 15:57:48 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 2024-03-10 15:57:48 proc_lsst.multi:201 [INFO] cancelling 16980071 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980071 on provider escience_a40 2024-03-10 15:57:49 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 2024-03-10 15:57:49 proc_lsst.multi:201 [INFO] cancelling 16980072 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980072 on provider escience_a40 2024-03-10 15:57:49 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro 2024-03-10 15:57:49 proc_lsst.multi:201 [INFO] cancelling 16980045 on provider astro proc_lsst.multi INFO: cancelling 16980045 on provider astro 2024-03-10 15:57:49 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro 2024-03-10 15:57:49 proc_lsst.multi:201 [INFO] cancelling 16980029 on provider astro proc_lsst.multi INFO: cancelling 16980029 on provider astro 2024-03-10 15:57:49 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro 2024-03-10 15:57:49 proc_lsst.multi:201 [INFO] cancelling 16980050 on provider astro proc_lsst.multi INFO: cancelling 16980050 on provider astro 2024-03-10 15:57:49 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro 2024-03-10 15:57:49 proc_lsst.multi:201 [INFO] cancelling 16980044 on provider astro proc_lsst.multi INFO: cancelling 16980044 on provider astro 2024-03-10 15:57:49 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro 2024-03-10 15:57:49 proc_lsst.multi:201 [INFO] cancelling 16980046 on provider astro proc_lsst.multi INFO: cancelling 16980046 on provider astro 2024-03-10 15:57:50 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k 2024-03-10 15:57:50 proc_lsst.multi:201 [INFO] cancelling 16980062 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980062 on provider escience_rtx6k 2024-03-10 15:57:50 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 2024-03-10 15:57:50 proc_lsst.multi:201 [INFO] cancelling 16980073 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980073 on provider escience_a40 2024-03-10 15:57:50 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 2024-03-10 15:57:50 proc_lsst.multi:201 [INFO] cancelling 16980078 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980078 on provider escience_a40 2024-03-10 15:57:50 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro 2024-03-10 15:57:50 proc_lsst.multi:201 [INFO] cancelling 16980051 on provider astro proc_lsst.multi INFO: cancelling 16980051 on provider astro 2024-03-10 15:57:50 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro 2024-03-10 15:57:50 proc_lsst.multi:201 [INFO] cancelling 16980054 on provider astro proc_lsst.multi INFO: cancelling 16980054 on provider astro 2024-03-10 15:57:51 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 2024-03-10 15:57:51 proc_lsst.multi:201 [INFO] cancelling 16980074 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980074 on provider escience_a40 2024-03-10 15:57:51 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k 2024-03-10 15:57:51 proc_lsst.multi:201 [INFO] cancelling 16980069 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980069 on provider escience_rtx6k 2024-03-10 15:57:51 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro 2024-03-10 15:57:51 proc_lsst.multi:201 [INFO] cancelling 16980053 on provider astro proc_lsst.multi INFO: cancelling 16980053 on provider astro 2024-03-10 15:57:51 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro 2024-03-10 15:57:51 proc_lsst.multi:201 [INFO] cancelling 16980037 on provider astro proc_lsst.multi INFO: cancelling 16980037 on provider astro 2024-03-10 15:57:51 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k 2024-03-10 15:57:51 proc_lsst.multi:201 [INFO] cancelling 16980060 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980060 on provider escience_rtx6k 2024-03-10 15:57:51 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k 2024-03-10 15:57:51 proc_lsst.multi:201 [INFO] cancelling 16980059 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980059 on provider escience_rtx6k 2024-03-10 15:57:52 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro 2024-03-10 15:57:52 proc_lsst.multi:201 [INFO] cancelling 16980034 on provider astro proc_lsst.multi INFO: cancelling 16980034 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-10 15:57:52 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-10 15:57:52 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16980029', '16980030', '16980031', '16980032', '16980033', '16980034', '16980035', '16980036', '16980037', '16980038', '16980039', '16980040', '16980041', '16980042', '16980043', '16980044', '16980045', '16980046', '16980047', '16980048', '16980049', '16980050', '16980051', '16980052', '16980053', '16980054', '16980055', '16980056', '16980057', '16980058', '16980071', '16980072', '16980073', '16980074', '16980075', '16980076', '16980077', '16980078', '16980079', '16980080', '16980081', '16980059', '16980060', '16980061', '16980062', '16980069', '90235'] proc_lsst.multi INFO: new jobs since last cancel ['16980029', '16980030', '16980031', '16980032', '16980033', '16980034', '16980035', '16980036', '16980037', '16980038', '16980039', '16980040', '16980041', '16980042', '16980043', '16980044', '16980045', '16980046', '16980047', '16980048', '16980049', '16980050', '16980051', '16980052', '16980053', '16980054', '16980055', '16980056', '16980057', '16980058', '16980071', '16980072', '16980073', '16980074', '16980075', '16980076', '16980077', '16980078', '16980079', '16980080', '16980081', '16980059', '16980060', '16980061', '16980062', '16980069', '90235'] 2024-03-10 15:57:52 proc_lsst.multi:146 [INFO] found job 16980029 in provider astro proc_lsst.multi INFO: found job 16980029 in provider astro 2024-03-10 15:57:52 proc_lsst.multi:201 [INFO] cancelling 16980029 on provider astro proc_lsst.multi INFO: cancelling 16980029 on provider astro 2024-03-10 15:57:52 proc_lsst.multi:146 [INFO] found job 16980030 in provider astro proc_lsst.multi INFO: found job 16980030 in provider astro 2024-03-10 15:57:52 proc_lsst.multi:201 [INFO] cancelling 16980030 on provider astro proc_lsst.multi INFO: cancelling 16980030 on provider astro 2024-03-10 15:57:52 proc_lsst.multi:146 [INFO] found job 16980031 in provider astro proc_lsst.multi INFO: found job 16980031 in provider astro 2024-03-10 15:57:52 proc_lsst.multi:201 [INFO] cancelling 16980031 on provider astro proc_lsst.multi INFO: cancelling 16980031 on provider astro 2024-03-10 15:57:52 proc_lsst.multi:146 [INFO] found job 16980032 in provider astro proc_lsst.multi INFO: found job 16980032 in provider astro 2024-03-10 15:57:52 proc_lsst.multi:201 [INFO] cancelling 16980032 on provider astro proc_lsst.multi INFO: cancelling 16980032 on provider astro 2024-03-10 15:57:52 proc_lsst.multi:146 [INFO] found job 16980033 in provider astro proc_lsst.multi INFO: found job 16980033 in provider astro 2024-03-10 15:57:52 proc_lsst.multi:201 [INFO] cancelling 16980033 on provider astro proc_lsst.multi INFO: cancelling 16980033 on provider astro 2024-03-10 15:57:53 proc_lsst.multi:146 [INFO] found job 16980034 in provider astro proc_lsst.multi INFO: found job 16980034 in provider astro 2024-03-10 15:57:53 proc_lsst.multi:201 [INFO] cancelling 16980034 on provider astro proc_lsst.multi INFO: cancelling 16980034 on provider astro 2024-03-10 15:57:53 proc_lsst.multi:146 [INFO] found job 16980035 in provider astro proc_lsst.multi INFO: found job 16980035 in provider astro 2024-03-10 15:57:53 proc_lsst.multi:201 [INFO] cancelling 16980035 on provider astro proc_lsst.multi INFO: cancelling 16980035 on provider astro 2024-03-10 15:57:53 proc_lsst.multi:146 [INFO] found job 16980036 in provider astro proc_lsst.multi INFO: found job 16980036 in provider astro 2024-03-10 15:57:53 proc_lsst.multi:201 [INFO] cancelling 16980036 on provider astro proc_lsst.multi INFO: cancelling 16980036 on provider astro 2024-03-10 15:57:53 proc_lsst.multi:146 [INFO] found job 16980037 in provider astro proc_lsst.multi INFO: found job 16980037 in provider astro 2024-03-10 15:57:53 proc_lsst.multi:201 [INFO] cancelling 16980037 on provider astro proc_lsst.multi INFO: cancelling 16980037 on provider astro 2024-03-10 15:57:53 proc_lsst.multi:146 [INFO] found job 16980038 in provider astro proc_lsst.multi INFO: found job 16980038 in provider astro 2024-03-10 15:57:53 proc_lsst.multi:201 [INFO] cancelling 16980038 on provider astro proc_lsst.multi INFO: cancelling 16980038 on provider astro 2024-03-10 15:57:54 proc_lsst.multi:146 [INFO] found job 16980039 in provider astro proc_lsst.multi INFO: found job 16980039 in provider astro 2024-03-10 15:57:54 proc_lsst.multi:201 [INFO] cancelling 16980039 on provider astro proc_lsst.multi INFO: cancelling 16980039 on provider astro 2024-03-10 15:57:56 proc_lsst.multi:146 [INFO] found job 16980040 in provider astro proc_lsst.multi INFO: found job 16980040 in provider astro 2024-03-10 15:57:56 proc_lsst.multi:201 [INFO] cancelling 16980040 on provider astro proc_lsst.multi INFO: cancelling 16980040 on provider astro 2024-03-10 15:57:56 proc_lsst.multi:146 [INFO] found job 16980041 in provider astro proc_lsst.multi INFO: found job 16980041 in provider astro 2024-03-10 15:57:56 proc_lsst.multi:201 [INFO] cancelling 16980041 on provider astro proc_lsst.multi INFO: cancelling 16980041 on provider astro 2024-03-10 15:57:56 proc_lsst.multi:146 [INFO] found job 16980042 in provider astro proc_lsst.multi INFO: found job 16980042 in provider astro 2024-03-10 15:57:56 proc_lsst.multi:201 [INFO] cancelling 16980042 on provider astro proc_lsst.multi INFO: cancelling 16980042 on provider astro 2024-03-10 15:57:57 proc_lsst.multi:146 [INFO] found job 16980043 in provider astro proc_lsst.multi INFO: found job 16980043 in provider astro 2024-03-10 15:57:57 proc_lsst.multi:201 [INFO] cancelling 16980043 on provider astro proc_lsst.multi INFO: cancelling 16980043 on provider astro 2024-03-10 15:57:57 proc_lsst.multi:146 [INFO] found job 16980044 in provider astro proc_lsst.multi INFO: found job 16980044 in provider astro 2024-03-10 15:57:57 proc_lsst.multi:201 [INFO] cancelling 16980044 on provider astro proc_lsst.multi INFO: cancelling 16980044 on provider astro 2024-03-10 15:57:57 proc_lsst.multi:146 [INFO] found job 16980045 in provider astro proc_lsst.multi INFO: found job 16980045 in provider astro 2024-03-10 15:57:57 proc_lsst.multi:201 [INFO] cancelling 16980045 on provider astro proc_lsst.multi INFO: cancelling 16980045 on provider astro 2024-03-10 15:57:57 proc_lsst.multi:146 [INFO] found job 16980046 in provider astro proc_lsst.multi INFO: found job 16980046 in provider astro 2024-03-10 15:57:57 proc_lsst.multi:201 [INFO] cancelling 16980046 on provider astro proc_lsst.multi INFO: cancelling 16980046 on provider astro 2024-03-10 15:57:57 proc_lsst.multi:146 [INFO] found job 16980047 in provider astro proc_lsst.multi INFO: found job 16980047 in provider astro 2024-03-10 15:57:57 proc_lsst.multi:201 [INFO] cancelling 16980047 on provider astro proc_lsst.multi INFO: cancelling 16980047 on provider astro 2024-03-10 15:57:57 proc_lsst.multi:146 [INFO] found job 16980048 in provider astro proc_lsst.multi INFO: found job 16980048 in provider astro 2024-03-10 15:57:57 proc_lsst.multi:201 [INFO] cancelling 16980048 on provider astro proc_lsst.multi INFO: cancelling 16980048 on provider astro 2024-03-10 15:57:58 proc_lsst.multi:146 [INFO] found job 16980049 in provider astro proc_lsst.multi INFO: found job 16980049 in provider astro 2024-03-10 15:57:58 proc_lsst.multi:201 [INFO] cancelling 16980049 on provider astro proc_lsst.multi INFO: cancelling 16980049 on provider astro 2024-03-10 15:57:58 proc_lsst.multi:146 [INFO] found job 16980050 in provider astro proc_lsst.multi INFO: found job 16980050 in provider astro 2024-03-10 15:57:58 proc_lsst.multi:201 [INFO] cancelling 16980050 on provider astro proc_lsst.multi INFO: cancelling 16980050 on provider astro 2024-03-10 15:57:58 proc_lsst.multi:146 [INFO] found job 16980051 in provider astro proc_lsst.multi INFO: found job 16980051 in provider astro 2024-03-10 15:57:58 proc_lsst.multi:201 [INFO] cancelling 16980051 on provider astro proc_lsst.multi INFO: cancelling 16980051 on provider astro 2024-03-10 15:57:58 proc_lsst.multi:146 [INFO] found job 16980052 in provider astro proc_lsst.multi INFO: found job 16980052 in provider astro 2024-03-10 15:57:58 proc_lsst.multi:201 [INFO] cancelling 16980052 on provider astro proc_lsst.multi INFO: cancelling 16980052 on provider astro 2024-03-10 15:57:58 proc_lsst.multi:146 [INFO] found job 16980053 in provider astro proc_lsst.multi INFO: found job 16980053 in provider astro 2024-03-10 15:57:58 proc_lsst.multi:201 [INFO] cancelling 16980053 on provider astro proc_lsst.multi INFO: cancelling 16980053 on provider astro 2024-03-10 15:57:58 proc_lsst.multi:146 [INFO] found job 16980054 in provider astro proc_lsst.multi INFO: found job 16980054 in provider astro 2024-03-10 15:57:58 proc_lsst.multi:201 [INFO] cancelling 16980054 on provider astro proc_lsst.multi INFO: cancelling 16980054 on provider astro 2024-03-10 15:57:59 proc_lsst.multi:146 [INFO] found job 16980055 in provider astro proc_lsst.multi INFO: found job 16980055 in provider astro 2024-03-10 15:57:59 proc_lsst.multi:201 [INFO] cancelling 16980055 on provider astro proc_lsst.multi INFO: cancelling 16980055 on provider astro 2024-03-10 15:57:59 proc_lsst.multi:146 [INFO] found job 16980056 in provider astro proc_lsst.multi INFO: found job 16980056 in provider astro 2024-03-10 15:57:59 proc_lsst.multi:201 [INFO] cancelling 16980056 on provider astro proc_lsst.multi INFO: cancelling 16980056 on provider astro 2024-03-10 15:57:59 proc_lsst.multi:146 [INFO] found job 16980057 in provider astro proc_lsst.multi INFO: found job 16980057 in provider astro 2024-03-10 15:57:59 proc_lsst.multi:201 [INFO] cancelling 16980057 on provider astro proc_lsst.multi INFO: cancelling 16980057 on provider astro 2024-03-10 15:57:59 proc_lsst.multi:146 [INFO] found job 16980058 in provider astro proc_lsst.multi INFO: found job 16980058 in provider astro 2024-03-10 15:57:59 proc_lsst.multi:201 [INFO] cancelling 16980058 on provider astro proc_lsst.multi INFO: cancelling 16980058 on provider astro 2024-03-10 15:57:59 proc_lsst.multi:146 [INFO] found job 16980071 in provider escience_a40 proc_lsst.multi INFO: found job 16980071 in provider escience_a40 2024-03-10 15:57:59 proc_lsst.multi:201 [INFO] cancelling 16980071 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980071 on provider escience_a40 2024-03-10 15:57:59 proc_lsst.multi:146 [INFO] found job 16980072 in provider escience_a40 proc_lsst.multi INFO: found job 16980072 in provider escience_a40 2024-03-10 15:57:59 proc_lsst.multi:201 [INFO] cancelling 16980072 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980072 on provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:146 [INFO] found job 16980073 in provider escience_a40 proc_lsst.multi INFO: found job 16980073 in provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:201 [INFO] cancelling 16980073 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980073 on provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:146 [INFO] found job 16980074 in provider escience_a40 proc_lsst.multi INFO: found job 16980074 in provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:201 [INFO] cancelling 16980074 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980074 on provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:146 [INFO] found job 16980075 in provider escience_a40 proc_lsst.multi INFO: found job 16980075 in provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:201 [INFO] cancelling 16980075 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980075 on provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:146 [INFO] found job 16980076 in provider escience_a40 proc_lsst.multi INFO: found job 16980076 in provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:201 [INFO] cancelling 16980076 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980076 on provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:146 [INFO] found job 16980077 in provider escience_a40 proc_lsst.multi INFO: found job 16980077 in provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:201 [INFO] cancelling 16980077 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980077 on provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:146 [INFO] found job 16980078 in provider escience_a40 proc_lsst.multi INFO: found job 16980078 in provider escience_a40 2024-03-10 15:58:00 proc_lsst.multi:201 [INFO] cancelling 16980078 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980078 on provider escience_a40 2024-03-10 15:58:01 proc_lsst.multi:146 [INFO] found job 16980079 in provider escience_a40 proc_lsst.multi INFO: found job 16980079 in provider escience_a40 2024-03-10 15:58:01 proc_lsst.multi:201 [INFO] cancelling 16980079 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980079 on provider escience_a40 2024-03-10 15:58:01 proc_lsst.multi:146 [INFO] found job 16980080 in provider escience_a40 proc_lsst.multi INFO: found job 16980080 in provider escience_a40 2024-03-10 15:58:01 proc_lsst.multi:201 [INFO] cancelling 16980080 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980080 on provider escience_a40 2024-03-10 15:58:01 proc_lsst.multi:146 [INFO] found job 16980081 in provider escience_a40 proc_lsst.multi INFO: found job 16980081 in provider escience_a40 2024-03-10 15:58:01 proc_lsst.multi:201 [INFO] cancelling 16980081 on provider escience_a40 proc_lsst.multi INFO: cancelling 16980081 on provider escience_a40 2024-03-10 15:58:01 proc_lsst.multi:146 [INFO] found job 16980059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980059 in provider escience_rtx6k 2024-03-10 15:58:01 proc_lsst.multi:201 [INFO] cancelling 16980059 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980059 on provider escience_rtx6k 2024-03-10 15:58:01 proc_lsst.multi:146 [INFO] found job 16980060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980060 in provider escience_rtx6k 2024-03-10 15:58:01 proc_lsst.multi:201 [INFO] cancelling 16980060 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980060 on provider escience_rtx6k 2024-03-10 15:58:01 proc_lsst.multi:146 [INFO] found job 16980061 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980061 in provider escience_rtx6k 2024-03-10 15:58:01 proc_lsst.multi:201 [INFO] cancelling 16980061 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980061 on provider escience_rtx6k 2024-03-10 15:58:02 proc_lsst.multi:146 [INFO] found job 16980062 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980062 in provider escience_rtx6k 2024-03-10 15:58:02 proc_lsst.multi:201 [INFO] cancelling 16980062 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980062 on provider escience_rtx6k 2024-03-10 15:58:02 proc_lsst.multi:146 [INFO] found job 16980069 in provider escience_rtx6k proc_lsst.multi INFO: found job 16980069 in provider escience_rtx6k 2024-03-10 15:58:02 proc_lsst.multi:201 [INFO] cancelling 16980069 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16980069 on provider escience_rtx6k 2024-03-10 15:58:02 proc_lsst.multi:146 [INFO] found job 90235 in provider local proc_lsst.multi INFO: found job 90235 in provider local 2024-03-10 15:58:02 proc_lsst.multi:201 [INFO] cancelling 90235 on provider local proc_lsst.multi INFO: cancelling 90235 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 90235 parsl.providers.local.local WARNING: Failed to kill PID: 90235 and child processes on local 2024-03-10 15:58:03 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 2079.4319 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190706_B0b_science#step1_20240310T152300Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 2079.4438 seconds; current memory usage: 0.371 Gibyte, delta: 0.006 Gibyte, peak delta: 0.006 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 2101.1466 seconds; current memory usage: 0.371 Gibyte, delta: 0.233 Gibyte, peak delta: 0.233 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.371 Gibyte (main), 0.371 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190706_B0b_science#step1_20240310T152300Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 35m4.474s user 5m0.653s sys 3m5.856s